Self-EmoQ: Plutchik-Guided Value-based Planning to Drive Streaming Emotional TTS
Pith reviewed 2026-07-05 09:43 UTC · model glm-5.2
The pith
RL Planner Picks System Emotion Before Speaking
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The core claim is that treating emotion as a planned action rather than a recognized label—selecting it before text generation via DQN on an LLM, with a hybrid reward of dataset imitation and GPT-4o-scored Plutchik theory—produces better emotional dialogue and streaming speech than prompting, supervised learning, or tabular Q-learning. The Q-values come from averaging LLM output token logprobs in a multiple-choice format, and the learned policy internalizes the adjacency structure of Plutchik's emotion wheel.
What carries the argument
Self-EmoQ: a DQN-trained LLM module where state = dialogue history + user query, action = discrete emotion label, reward = (1-w) * imitation_match + w * Plutchik_Score. Q(s,a) is estimated by averaging the logprobs of the LLM's output tokens when the emotion is presented as a multiple-choice option. The Plutchik Score is a 0-5 average across three dimensions (alignment, transition plausibility, function consistency) assigned by GPT-4o. The selected emotion then conditions both the response-generation LLM and the streaming emotional TTS.
If this is right
- If the approach generalizes, any conversational AI pipeline with streaming TTS could adopt pre-generation emotion planning without waiting for post-hoc emotion recognition, removing a latency bottleneck in real-time emotional speech.
- The logprob-as-Q-value technique could be applied to other discrete planning decisions in dialogue (strategy selection, intent selection, persona selection) without training a separate value network from scratch.
- The hybrid reward design—dataset imitation plus theory-driven scoring by a frontier LLM—suggests a general pattern for injecting domain knowledge into RL training when expert-annotated reward signals are expensive or unavailable.
- The observed alignment between learned transition matrices and the Plutchik wheel topology suggests the theory-driven reward genuinely shapes policy behavior, not just the metric being optimized.
Where Pith is reading between the lines
- The framework's success depends on GPT-4o being a reliable emotion-theory evaluator, but the validation is limited to 100 human-scored samples with correlation statistics; the gradient-level reliability of these scores during RL training is not directly assessed.
- If GPT-4o's Plutchik scoring is systematically biased (e.g., favoring certain emotion pairs regardless of context), the learned Q-values would optimize a proxy that diverges from genuine emotional appropriateness in ways the ranking metrics might not catch.
- The discrete emotion action space (7-10 labels depending on dataset) is a significant simplification; real emotional expression involves intensity, blending, and gradual shifts that a categorical action cannot capture, which the authors acknowledge as a limitation.
- The finding that pre-SFT degrades RL performance (ablation: w/ SFT) is worth noting—it suggests that supervised warm-starting on emotion labels can constrain exploration, which has implications for how emotion-planning modules should be initialized in practice.
Load-bearing premise
The theory-driven half of the training reward is generated by GPT-4o scoring emotional appropriateness on a 0-5 scale according to Plutchik's theory. If GPT-4o's scores are noisy, biased toward certain emotion transitions, or insensitive to the Plutchik wheel's topology, the entire learned policy optimizes a proxy that may not reflect genuine emotional appropriateness.
What would settle it
If replacing GPT-4o with a different LLM scorer (or human annotators at scale) as the Plutchik reward source produced substantially different learned policies or transition matrices, the framework's claims about theory-driven shaping would be undermined, since the policy would be revealed as fitting the scorer rather than the theory.
Figures
read the original abstract
Emotional interaction is increasingly crucial for conversational AI, yet current systems lack a self-emotion determination mechanism to drive the streaming text-to-speech (TTS) synthesis. We propose an emotion-planning framework that determines the emotion prior to the textual generation, grounding the downstream emotional TTS in a streaming manner. The framework is implemented by a plug-and-play LLM module, initialized from pretrained LLMs, and trained by reinforcement learning (RL) with emotions as the actions. A hybrid reward is employed which combines imitation signals with theory-driven scoring, in which the theory of Plutchik's wheel of emotions is adopted. By experiments on DailyDialog, EmoryNLP, IMEOCAP, and MELD, our method outperforms prompting and finetuning baselines on both emotion determination and response quality. We finally implement an entire streaming pipeline for real-time deployment, with the speech quality confirming the framework's emotional alignment, contextual coherence, and expressive fluency. Codes, cases, and demos are available in https://sixingdeguo.github.io/EmoQ-page/.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Self-EmoQ, an LLM-based DQN planner that selects the system emotion before response generation, using a hybrid reward combining dataset-label imitation and GPT-4o-scored Plutchik theory. The planned emotion then conditions both text generation and streaming emotional TTS. Experiments span four datasets (DailyDialog, EmoryNLP, MELD, IEMOCAP), seven baselines, ablations, human evaluation, and downstream TTS quality. The system addresses a real architectural gap in streaming emotional conversational AI: ERC-based methods cannot provide emotion labels until after text generation, which is incompatible with streaming TTS. The approach of treating emotion as a planned action variable via value-based RL on an LLM is well-motivated, and the Plutchik-theory-grounded reward is a thoughtful design choice. Code and demos are publicly available, which is a strength.
Significance. The paper makes a practical contribution to the integration of emotional planning with streaming TTS pipelines. The framing of emotion as a sequential decision variable rather than a reactive label is a meaningful architectural contribution for the streaming setting. The Plutchik-guided reward design, the transition-matrix analysis (Figure 6), and the ratio-of-expected-transitions validation (Figure 7) provide theory-grounded diagnostics beyond standard benchmark numbers. The public code and demo page, the downstream TTS evaluation with SpeechBERTScore and PESQ, and the human evaluation across multiple dimensions are all commendable. However, the significance of the emotion-determination results is substantially tempered by a circularity concern in the evaluation methodology (see Major Comment 1), which must be resolved for the central claim to be credible.
major comments (2)
- §4.2, Table 3: The evaluation methodology for emotion determination is circular. The text states rankings are 'compared against the corresponding reward signals' to compute Recall@K, NDCG, and MRR. The reward signals are the hybrid reward from Eq.(10), which is the same quantity the RL training optimizes. If these ranking metrics use the hybrid reward as the relevance signal, then Table 3 measures how well learned Q-values reproduce the training objective's own ranking, not whether the model identifies genuinely appropriate emotions. The ablation in Table 6 provides ambiguous evidence: 'w/o history' drops Reward from 0.57 to 0.21 on DailyDialog while Recall@3 stays at 0.82, suggesting Recall may use dataset labels as ground truth — but NDCG drops (0.92 to 0.88) and MRR drops (0.72 to 0.48), leaving the methodology unclear. The authors must clarify exactly what ground-truth signal is used
- Table 3, 'Reward' column: Reporting the hybrid reward from Eq.(10) as an evaluation metric alongside Recall/NDCG/MRR is partially circular, since this is the training objective. The independent metrics (Recall, NDCG, MRR) would carry the evaluation if they use an independent ground truth (dataset labels), but this is not clearly stated. The authors should either remove the Reward column from the main results table or explicitly label it as a training-objective diagnostic, not an independent evaluation metric.
minor comments (9)
- Dataset name inconsistency: the paper uses both 'IMEOCAP' and 'IEMOCAP' (e.g., abstract and Table 2 say 'IMEOCAP'; the dataset description in Appendix A.5 says 'IEMOCAP'). Please standardize to 'IEMOCAP.'
- Conclusion: 'play-and-plug' should be 'plug-and-play.'
- Table 2 caption reads 'Statistics of datasets and evaluation metrics,' but the table contains no evaluation metrics — only dataset statistics. Please correct the caption.
- §3.4, Eq.(13): The text says Q-values are obtained by 'the average of action logits,' but the equation uses LLM output. Please clarify whether logprobs or raw logits are averaged (these are different quantities).
- Figure 5: The x-axis is labeled with gamma values (0.5 to 0.95), but §4.1 states gamma=0.8. The sensitivity study should clarify which datasets use which gamma in the final reported results, and whether the optimal gamma from Figure 5 was selected post-hoc.
- §4.4: The text references 'Figure 6' for training loss curves, but Figure 6 shows emotional transition matrices. The loss curves appear to be in Figure 4. Please correct the cross-reference.
- Appendix B.2, Table 11: The dimension labels in the table ('Alignment,' 'Emotion,' 'Effectiveness') do not match the three scoring dimensions defined in the prompt and §3.2 ('Emotion Alignment,' 'Emotion Transition Plausibility,' 'Emotion-Function Consistency'). Please align the labels.
- §4.5: The CoCT baseline is introduced here for the first time but was not included in the main emotion-determination or response-generation experiments (Tables 3-4). Please either include CoCT in the earlier tables or explain why it is only applicable to the TTS comparison.
- Algorithm 1, line 7: 'Form state s_t = (h_t, s^u_t)' uses 's^u_t' but the user utterance is defined as 'x^u_t' in Eq.(8). Please use consistent notation.
Simulated Author's Rebuttal
We thank the referee for the careful reading and the constructive comments. Both major comments concern the evaluation methodology in §4.2 and Table 3, and we agree that the current presentation is unclear. Below we address each comment point by point.
read point-by-point responses
-
Referee: §4.2, Table 3: The evaluation methodology for emotion determination is circular. The text states rankings are 'compared against the corresponding reward signals' to compute Recall@K, NDCG, and MRR. The reward signals are the hybrid reward from Eq.(10), which is the same quantity the RL training optimizes. If these ranking metrics use the hybrid reward as the relevance signal, then Table 3 measures how well learned Q-values reproduce the training objective's own ranking, not whether the model identifies genuinely appropriate emotions. The ablation in Table 6 provides ambiguous evidence: 'w/o history' drops Reward from 0.57 to 0.21 on DailyDialog while Recall@3 stays at 0.82, suggesting Recall may use dataset labels as ground truth — but NDCG drops (0.92 to 0.88) and MRR drops (0.72 to 0.48), leaving the methodology unclear. The authors must clarify exactly what ground-truth signal is used
Authors: We thank the referee for identifying this ambiguity, which stems from imprecise wording in §4.2. To clarify directly: Recall@K, NDCG, and MRR are computed against the **dataset ground-truth emotion labels**, not the hybrid reward. The formal metric definitions in Appendix A.3 already reflect this — for instance, Recall@K uses G defined as 'the set of relevant (ground-truth) emotion labels,' and NDCG uses graded relevance derived from these labels. The phrase 'compared against the corresponding reward signals' in §4.2 is a writing error; it should read 'compared against the corresponding ground-truth emotion labels.' We will correct this in the revised manuscript. The ablation evidence the referee cites is in fact consistent with this interpretation: when dialogue history is removed (w/o history), the hybrid Reward drops sharply (0.57→0.21 on DailyDialog) because the Plutchik component of the reward degrades without contextual information, but Recall@3 remains at 0.82 because the dataset labels are unchanged and the model can still identify the correct emotion within the top-3 candidates. The MRR drop (0.72→0.48) reflects degraded ranking quality — the correct emotion remains in the top-3 but is ranked lower (e.g., 3rd instead of 1st), which reduces reciprocal rank without affecting Recall@3. NDCG's smaller drop (0.92→0.88) is consistent with this explanation, as NDCG is less sensitive to rank-position shifts when the relevant item remains highly ranked. We will revise §4.2 to state unambiguously that the ranking metrics use dataset ground-truth labels as the relevance signal, and we will add a sentence explaining the ablation pattern the referee identified. revision: yes
-
Referee: Table 3, 'Reward' column: Reporting the hybrid reward from Eq.(10) as an evaluation metric alongside Recall/NDCG/MRR is partially circular, since this is the training objective. The independent metrics (Recall, NDCG, MRR) would carry the evaluation if they use an independent ground truth (dataset labels), but this is not clearly stated. The authors should either remove the Reward column from the main results table or explicitly label it as a training-objective diagnostic, not an independent evaluation metric.
Authors: The referee is correct that the Reward column reports the training objective and is therefore not an independent evaluation metric. As clarified in our response to Major Comment 1, the independent evaluation is carried by Recall@K, NDCG, and MRR, which use dataset ground-truth labels. We agree that the Reward column should not be presented on equal footing with these independent metrics. In the revised manuscript, we will explicitly label the Reward column as 'Training-Objective Diagnostic' in the table caption and add a note in §4.2 stating that this column reflects the hybrid reward being optimized during RL training and should not be interpreted as an independent evaluation metric. We will retain it in the table because it provides useful diagnostic information about convergence and the relative contribution of the Plutchik component (as evidenced by the ablation analysis), but we will make its status unambiguous. revision: yes
Circularity Check
Reward column in Table 3 is the training objective from Eq.(10), creating partial circularity; ranking metrics (Recall/NDCG/MRR) appear to use ground-truth labels and are not circular.
specific steps
-
fitted input called prediction
[Section 4.2, Table 3 'Reward' column]
"We evaluate the quality of emotion decisions by ranking candidate emotions according to the Q-values. The resulting rankings are then compared against the corresponding reward signals to compute ranking-based metrics."
The 'Reward' column in Table 3 reports the hybrid reward r_t from Eq.(10), which is the exact quantity the DQN training (Algorithm 1, line 9: 'Obtain x_s_t, r_t by Eq(10)') optimizes via the Bellman loss in Eq.(3). Reporting this as an evaluation metric alongside independent metrics like Recall/NDCG/MRR creates partial circularity: the model is evaluated partly on the objective it was trained to maximize. However, the ranking metrics themselves (Recall@K, NDCG, MRR) are defined in Appendix A.3 using ground-truth labels G (Eq. 15: 'Recall@K = |R_K ∩ G| / |G|'), not the reward signal. The ablation in Table 6 provides supporting evidence: 'w/o history' drops Reward from 0.57 to 0.21 while Recall@3 stays at 0.82, confirming Recall uses dataset labels, not the reward. The circularity is thus局限于
full rationale
The paper has one partially circular element: the 'Reward' column in Table 3 directly reports the training objective from Eq.(10) as an evaluation metric. This is a fitted-input-called-prediction pattern, but it is a single column among multiple independent metrics. The ranking metrics (Recall, NDCG, MRR) are defined against ground-truth labels (Appendix A.3, Eqs. 15-19), not against the reward signal, and the ablation results confirm this. The response generation metrics (BLEU, ROUGE-L, CIDEr, Distinct-2), human evaluation, and downstream TTS metrics are all independent of the training reward. The self-citation to Wang et al. (2025b) for the Q-value estimation method (Eq. 13) is a methodological reference, not a load-bearing uniqueness claim. The GPT-4o Plutchik scoring is validated with a human correlation study (Table 11, ρ=0.87-0.96), providing external evidence. The core claims rest on independent benchmarks, so the circularity is limited and partial.
Axiom & Free-Parameter Ledger
free parameters (7)
- w (Plutchik score weight) =
varies by dataset (0.3-0.9, see Table 12)
- gamma (discount factor) =
0.8 (default), tuned per dataset (0.5-0.95, see Figure 5)
- epsilon (exploration rate) =
0.1
- lr (learning rate) =
1e-5
- C (target update interval) =
5
- L (context length) =
1024
- B (batch size) =
512
axioms (4)
- domain assumption Plutchik's Wheel of Emotion (1982) provides a valid and sufficient taxonomy for emotional categories, transitions, and behavioral functions in dialogue.
- domain assumption GPT-4o can reliably score emotional alignment, transition plausibility, and emotion-function consistency according to Plutchik theory.
- ad hoc to paper Averaging output token logprobs of an LLM yields a valid Q-value approximation for discrete action selection.
- domain assumption The utterance-level MDP formulation (state = dialogue history + user query, action = system emotion) satisfies the Markov property.
Reference graph
Works this paper leans on
-
[1]
Making Flow-Matching-Based Zero-Shot Text-to-Speech Laugh as You Like
Exploring perception uncertainty for emotion recognition in dyadic conversation and music listen- ing.Cognitive Computation, 13(2):231–240. Kalervo Järvelin and Jaana Kekäläinen. 2002. Cu- mulated gain-based evaluation of ir techniques. ACM Transactions on Information Systems (TOIS), 20(4):422–446. Xincheng Ju, Dong Zhang, Suyang Zhu, Junhui Li, Shoushan ...
work page internal anchor Pith review Pith/arXiv arXiv 2002
-
[2]
A Diversity-Promoting Objective Function for Neural Conversation Models
Msemotts: Multi-scale emotion transfer, pre- diction, and control for emotional speech synthesis. IEEE/ACM Transactions on Audio, Speech, and Lan- guage Processing, 30:853–864. Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2015. A diversity-promoting objec- tive function for neural conversation models.arXiv preprint arXiv:1510.030...
work page internal anchor Pith review Pith/arXiv arXiv 2015
-
[3]
Ece-tts: A zero-shot emotion text-to-speech model with simplified and precise control.Applied Sciences, 15(9):5108. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. InText summarization branches out, pages 74–81. Navonil Majumder, Soujanya Poria, Devamanyu Haz- arika, Rada Mihalcea, Alexander Gelbukh, and Erik Cambria. 2019. Dia...
work page internal anchor Pith review Pith/arXiv arXiv 2004
-
[4]
Coe: A clue of emotion framework for emo- tion recognition in conversations. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 23548–23563. Haoxiang Shi, Ziqi Liang, and Jun Yu. 2024. Emotional cues extraction and fusion for multi-modal emotion prediction and recognition in conversatio...
work page internal anchor Pith review Pith/arXiv arXiv 2024
-
[5]
Xiao Sun, Chen Zhang, and Lian Li
Dynamic emotional transition sampling and emotional guidance of individuals based on conver- sation.IEEE Transactions on Computational Social Systems, 11(1):1192–1204. Xiao Sun, Chen Zhang, and Lian Li. 2019. Dynamic emotion modelling and anomaly detection in conver- sation based on emotional transition tensor.Informa- tion Fusion, 46:11–22. Ramakrishna V...
-
[6]
Yunhe Xie, Yifan Liu, Chengjie Sun, and Shaobo Li
IEEE. Yunhe Xie, Yifan Liu, Chengjie Sun, and Shaobo Li
-
[7]
Towards Interpretable Mental Health Analysis with Large Language Models
Pseudo-utterance-guided contrastive network for emotion forecasting in conversations.Expert Systems with Applications, 279:127382. Kailai Yang, Shaoxiong Ji, Tianlin Zhang, Qianqian Xie, Ziyan Kuang, and Sophia Ananiadou. 2023. To- wards interpretable mental health analysis with large language models.arXiv preprint arXiv:2304.03347. Qing Yang, Zhenghao Li...
work page internal anchor Pith review Pith/arXiv arXiv 2023
-
[8]
Each emotion is associated with typical behaviors and functions: -J oy: Courting, mating; Reproduction -T rust: Grooming, sharing; Affiliation -F ear: Running, or flying away;Protection -Surprise: Stopping, alerting; Orientation -Sadness: Crying for help; Reintegration -Disgust: V omiting, pushing away; Rejection -Anger: Biting, hitting; Destruction - Ant...
-
[9]
- Transitions betweenoppositeemotions are less plausible unless mediated
Emotional transitions follow structured relationships: - Transitions betweenadjacentemotions are generally more natural. - Transitions betweenoppositeemotions are less plausible unless mediated. - Emotional responses should not abruptly contradict the user’s emotional state. Your task is to evaluate the system response ac- cording to these principles. His...
work page 1982
-
[10]
Emotion Alignment: To what extent does the response clearly express the target emotional state?
-
[11]
Emotion Transition Plausibility: Is the emotional transition from the user’s emo- tion to the system’s target emotion reasonable according to Plutchik’s emotional structure?
-
[12]
You wanna go over to Joey and Chandler’s?
Emotion–Function Consistency: Does the response exhibit the typical behavioral function associated with the target emotion? Your response needs to follow the following format: {alignment:int, transition:int, f unction:int,} A.2 Typical Emotion-Behavior Transitions The theory ofPlutchik’s Wheel of Emotions (Plutchik, 1982) identifies eight primary emotions...
work page 1982
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.