Pith. sign in

REVIEW 4 major objections 5 minor 65 references

AuEmoChat claims that conversational speech synthesis becomes more expressive and natural when emotion is represented as a learned discrete token space instead of a fixed set of categories, and when redundant dialogue context is merged usin

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 01:53 UTC pith:FOZUZ654

load-bearing objection The engineering is plausible and the token-merging/flow-matching combination is new, but the 'authentic emotion' result rests on a metric computed by the same codec the model is trained to satisfy — treat the headline claim as unproven. the 4 major comments →

arxiv 2607.15755 v2 pith:FOZUZ654 submitted 2026-07-17 cs.SD cs.AI

AuEmoChat: Authentic Emotion Understanding and Rendering for Conversational Speech Synthesis

Zhenqi Jia , Yuan Zhao , Aruukhan , Rui Liu , Haizhou Li This is my paper
classification cs.SD cs.AI
keywords conversational speech synthesisemotion tokenizationfinite scalar quantizationtoken mergingflow matchingauthentic emotiondialogue context modelingemotion perception
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that conversational speech synthesis is held back by two design choices: using a small set of predefined emotion categories (anger, happiness, etc.) as the only emotional control, and feeding the model every token from the multi-turn dialogue history. The authors propose AuEmoChat, which replaces the emotion label space with a learned discrete 'authentic emotion' token space obtained by quantizing emotional speech into tokens that reconstruct multi-axis perceived emotion scores, and which merges redundant text and speech tokens in the dialogue history guided by those emotion tokens. They then predict the target utterance's emotion token and speech tokens from the merged context, and render speech with a flow-matching model conditioned on the merged context, the emotion token, and an emotion classifier that steers the generated mel-spectrogram. On the NCSSD-EmCap dataset, AuEmoChat reports higher naturalness and emotion-consistency scores than existing CSS baselines, along with lower word error rate and mel distortion. If correct, the work shows that emotion can be treated as a learnable, continuous-ish discrete code rather than a fixed category set, and that compressing dialogue context is beneficial, not just efficient.

Core claim

AuEmoChat's central claim is that authentic emotional expression in conversational speech can be captured by a discrete emotion token space learned from speech itself, not by a predefined list of emotion labels. The tokenizer, AuEmoCodec, uses finite scalar quantization to map an emotion representation into a codebook of 1,000 codes (750 active), and is trained not to reconstruct the waveform but to reconstruct an LLM judge's perceived scores on seven basic-emotion axes. AuEmoToMe then merges redundant text and speech tokens in the dialogue history, weighting merges by similarity to the utterance's AuEmo token, so the compressed context preserves affective cues. An autoregressive model predi

What carries the argument

The load-bearing mechanism is AuEmoCodec, a finite-scalar-quantization autoencoder that converts emotional speech into a small set of discrete 'AuEmo' tokens by training to reconstruct multi-axis perceived emotion scores (produced by a large language model judge) rather than the audio waveform. This defines the authentic emotion token space that the rest of the system uses: AuEmoToMe uses each utterance's AuEmo token as an anchor to merge redundant context tokens, and the flow-matching renderer uses the predicted target AuEmo token as a conditioning signal and as the target for classifier guidance.

Load-bearing premise

The load-bearing premise is that a commercial LLM's perceived emotion scores on seven axes are a valid proxy for human emotion perception — a premise the paper does not independently verify, and one that its authenticity metric (computed with the same trained AuEmoCodec) could simply echo.

What would settle it

Collect human perceived-emotion ratings on the seven axes for a sample of NCSSD-EmCap utterances, train/evaluate AuEmoCodec against those human scores, and compare AuEmoACC's token agreement with human-rated similarity. If the token space reconstructed from the LLM judge does not match human ratings better than chance — or if a model trained on human scores performs no better than the baselines — the authenticity claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Emotion in CSS can be modeled as a dense, discrete code learned from speech rather than as one of seven fixed categories, capturing fine-grained states like excitement or tearful joy that share a basic label.
  • Moderately compressing the dialogue history (roughly 30% of tokens merged) removes distraction and improves both emotion accuracy and pronunciation quality, so longer conversations can be handled without losing context.
  • The Authentic Emotion Flow Matching with AuEmo classifier guidance (active after t=0.7) makes the transport path follow both the acoustic prior and the target emotion, improving context-consistent expression.
  • On the NCSSD-EmCap benchmark, the full system outperforms existing CSS baselines across all reported metrics — naturalness, emotion consistency, word error rate, mel distortion, speaker similarity, and both emotion accuracy measures.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The 'authentic' label depends on a validity link that the paper does not independently establish: the ground truth is a commercial LLM's perceived emotion scores (§4.4), and the authenticity metric (AuEmoACC) reuses the same trained AuEmoCodec (§5.3). Until human ratings confirm the same token-space geometry, 'authentic' is best read as 'aligned with the LLM judge.'
  • The approach suggests a general recipe for other synthesis domains: replace discrete categorical controls with learned, quantized latent codes trained to reconstruct dense perceptual attributes, then use those codes as anchors for context compression and as classifier-guidance targets.
  • Because the emotion token space has only 750 active codes, one could test whether particular tokens correspond to interpretable emotion mixtures or clusters (e.g., valence/arousal regions); if tokens are interpretable, they could form a bridge between categorical and dimensional emotion models.
  • The merging strategy is intra-modal only; a natural extension is cross-modal token merging, where redundant text and speech tokens referring to the same content are merged, which could be more efficient and perhaps more accurate on longer conversations.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes AuEmoChat, a conversational speech synthesis framework whose central claim is that it learns an 'authentic emotion' token space beyond Ekman categories and uses it to improve both expressiveness and speech quality. The architecture has three main components: (1) AuEmoCodec, an FSQ-based codec that maps emotional speech to discrete AuEmo tokens by reconstructing Gemini-2.5-Flash's perceived scores on seven emotion axes (§4.4); (2) AuEmoToMe, an emotion-guided token merging algorithm for compressing multimodal dialogue history while preserving emotion-relevant tokens (§4.2); and (3) Authentic Emotion Flow Matching, a flow-matching renderer conditioned on merged context, the predicted AuEmo token, and acoustic priors, with AuEmo-codec-based classifier guidance (§4.3, Eq. 18). The system is evaluated on NCSSD-EmCap against BaseCSS, ECSS, GPT-Talker, and Chain-Talker, with subjective N-DMOS/E-DMOS and objective WER/MCD/SpkSIM/EmoACC/AuEmoACC metrics. The paper reports consistent improvements on all metrics and a full ablation study supporting each component.

Significance. If the central claim were established, the contribution would be meaningful: a discrete, learned emotion-code space that goes beyond Ekman categories, combined with token merging and flow-matching rendering, is a plausible direction for conversational TTS. The engineering results are credible as incremental advances: the reported WER, MCD, and SpkSIM improvements over strong baselines are substantial, the ablations are systematic, and the authors state that code and demos will be released, which aids reproducibility. However, the paper's distinctive claim — 'authentic emotion understanding and rendering' — is not yet supported. The AuEmo token space is trained on Gemini-2.5-Flash pseudo-labels, the objective AuEmoACC metric is computed with the same AuEmoCodec that defines the token space, and the flow-matching objective explicitly optimizes toward that same codec via classifier guidance. This is a self-consistency loop. The human E-DMOS measures expressiveness/consistency with context, not authenticity of the learned emotion tokens. As a result, the paper currently demonstrates a well-engineered system with improved standard metrics, but the headline 'authentic emotion' claim requ

major comments (4)
  1. [§4.4, §5.3, Eq. (18)] The 'authentic emotion' evidence is circular. AuEmoCodec is trained to reconstruct Gemini-2.5-Flash's seven-axis perceived scores (§4.4), and is then used as (a) the AuEmo tokenizer for the dialogue history, (b) the classifier guidance in Eq. (18), and (c) the AuEmoACC evaluation metric (§5.3). The model is explicitly optimized to make generated speech map to the target AuEmo token under this codec, and then scored by the same codec. The AuEmoACC margin (28.71 vs. 24.12 over Chain-Talker) could therefore reflect codec-specific shortcuts or acoustic artifacts associated with token classes rather than human-perceived authenticity. This is load-bearing for the central claim. The authors should add human validation of the emotion token space — e.g., listener judgments of perceived emotion, mapping of tokens to emotion descriptions, or an independent emotion-recognition model not derived from
  2. [§4.4 and absent Appendix A] The supervision for AuEmoCodec is Gemini-2.5-Flash's perceived scores, but the prompt is 'provided in Appendix A,' which is not present in the manuscript. Without the prompt, the score scale, the possible range on each of the seven axes, and the instruction given to the judge model cannot be assessed. More importantly, no human validation of these pseudo-labels is reported: no correlation with human emotion ratings, no inter-rater agreement, no evidence that Gemini's perceived scores correspond to the 'authentic human emotion' the paper claims. Table 3's comparison of judge models only shows which LLM's labels are easiest for AuEmoCodec to reconstruct on its own training metric; it does not establish that Gemini-2.5-Flash has 'stronger authentic emotion perception ability.' Please provide the prompt, the score distributions, and human-agreement analysis.
  3. [§5.2] The baseline comparison is underspecified. The paper states that 'for fair comparison, all baseline models are configured to use the authentic emotion token space.' This means the reported baselines are not the published BaseCSS/ECSS/GPT-Talker/Chain-Talker systems; they are modified versions. The nature of these modifications is not described, and it is unclear whether injecting AuEmo tokens into baselines helps or hurts them. If the modified baselines are weakened by the token space, the comparison inflates AuEmoChat's advantage; if strengthened, the comparison is still not against the systems cited as SOTA. Please report unmodified baseline numbers as well, or describe the modification in enough detail that its effect can be judged.
  4. [§4.1, §6.3] The AuEmo token itself remains an unvalidated invented entity. A codebook of size 1000 is chosen, 750 tokens are activated, and the paper reports usage and reconstruction metrics, but there is no analysis of what individual tokens mean, how they relate to emotion categories or dimensions, whether distinct speakers/emotions map to distinct tokens, or whether the token space is stable across runs. Since the entire authenticity claim rests on this token space, the authors should provide token-level interpretability analysis, e.g., nearest-emotion-label statistics, token confusion patterns, or examples where the same utterance is tokenized under different conditions. Without such evidence, 'AuEmo token' is only a quantized embedding of Gemini labels.
minor comments (5)
  1. [§5.3, Table 1] The phrase 'significantly outperforms' is used in the abstract and §1, but no significance tests are reported for the subjective or objective metrics. The 95% confidence intervals for N-DMOS/E-DMOS are useful; a paired significance test or effect-size reporting would strengthen the claim.
  2. [§4.3, Eq. (18)] The activation threshold r=0.7 for classifier guidance is a free parameter. No sensitivity analysis is provided for this threshold, although it may materially affect the emotion/speech-quality trade-off. A short sweep would be helpful.
  3. [§7, absent Appendix D] The conclusion says 'More detailed limitations and future works are provided in Appendix D,' but no appendix is included. Since the manuscript already identifies a limitation statement, please include it; absent appendices also affect the reproducibility of the Gemini annotation prompt and implementation details.
  4. [§6.4, Fig. 3] The radar-style figure with five metrics on different scales is hard to read, and the plotted values are not all labeled. A table of the merging-rate sweep would be clearer and would allow readers to verify the claimed optimum at 30%.
  5. [§2.2] The claim 'Plutchik points out that humans can express about 34,000 distinct emotions' is an unusual and possibly inaccurate characterization of Plutchik's model. Please verify the source and either quote it precisely or soften the claim.

Circularity Check

2 steps flagged

The headline authenticity claim is measured by the same AuEmoCodec that the model is optimized to satisfy (Eq. 18 vs. AuEmoACC), and the judge-model choice is validated against the judge's own labels—so the 'authentic emotion' advantage is partly a self-consistency loop.

specific steps
  1. fitted input called prediction [§4.3 Eq. (18); §4.4; §5.3; §6.1]
    "we further introduce an AuEmo classifier guidance mechanism. Specifically, an auxiliary AuEmo classifier ψ is applied to the intermediate state φ_t(x) to estimate the probability that the current mel state matches the target AuEmo token f^E_N. ... ψ denotes the pre-trained AuEmoCodec. ... Authentic Emotion Accuracy (AuEmoACC) measures consistency in the authentic emotion token space using the trained AuEmoCodec."

    The same fitted AuEmoCodec defines the AuEmo token space, provides the classifier guidance in Eq. (18), and is then used as the AuEmoACC evaluator in §5.3. The flow-matching model is explicitly optimized to make intermediate mel states be classified as the target AuEmo token by AuEmoCodec, and the reported 'authentic emotion' improvement is scored by that same codec. The AuEmoACC margin (24.12 to 28.71) is therefore partly forced by the optimization target rather than by independent human-perceived authenticity; E-DMOS does not validate the semantic content of the learned token space.

  2. fitted input called prediction [§5.3 'AuEmoCodec Analysis Metrics'; §6.3 Table 3]
    "Tolerance at m axes (Tol@m) evaluates the reconstruction accuracy of multi-axis perceived emotion scores while allowing at most m mismatched axes. ... Gemini-2.5-Flash achieves the best performance on Tol@1, S-EAA, and D-EAA, demonstrating stronger authentic emotion perception ability."

    The ground-truth perceived scores GT^{ae}_i used to train and evaluate AuEmoCodec are Gemini-2.5-Flash's own annotations (§4.4). Tol@m, S-EAA, and D-EAA therefore measure how accurately AuEmoCodec reconstructs Gemini's own labels, not how well the judge agrees with human emotion perception. Selecting Gemini because it scores best on metrics whose reference labels come from Gemini itself is a self-consistency check; without human validation or inter-rater agreement, the conclusion that Gemini provides 'more accurate emotion supervision' is not externally anchored.

full rationale

The paper's speech-quality and some expressiveness results (N-DMOS, E-DMOS, WER, MCD, SpkSIM) are externally anchored and would support a narrower claim about naturalness and basic emotional expressiveness. However, the distinctive 'authentic emotion' contribution rests on AuEmoCodec tokens and AuEmoACC. That metric and the guidance objective share the same fitted model, so the central authenticity comparison is internally defined rather than independently tested. Two concrete self-referential steps support this: (1) AuEmoCodec is both the classifier-guidance model in Eq. (18) and the AuEmoACC evaluator in §5.3, making the AuEmoACC gain partly a construction artifact of optimizing toward that codec; (2) the judge-model analysis in §6.3 selects Gemini-2.5-Flash using metrics whose ground-truth labels are Gemini's own perceived scores, so it does not establish human-level 'authentic emotion perception.' The Gemini prompt is also deferred to an absent Appendix A, and no inter-rater agreement with human listeners is reported. No load-bearing self-citation circularity was found; the issue is self-consistency of the authenticity metric, not citation practices. Score 6 reflects a partial circularity: the authenticity claim is not fully forced, but its headline evidence reduces in part to the model's own fitted codec.

Axiom & Free-Parameter Ledger

3 free parameters · 6 axioms · 1 invented entities

The central novelty rests on three unvalidated premises: (i) the LLM judge supplies valid emotion ground truth, (ii) seven-axis perceived scores span 'authentic' emotion, and (iii) the AuEmoCodec is a valid evaluator of its own token space. These push the authenticity claim toward circularity. The engineering components otherwise rely on standard FSQ, ToMe, and flow-matching math.

free parameters (3)
  • FSQ quantization levels / emotion codebook size = [8,5,5,5] (1,000 codes; 750 active after training)
    Chosen from Table 3's comparative study of quantization paradigms and codebook capacity using Tol@1, S-EAA, and D-EAA on the evaluation dataset.
  • Token merging ratio ρ = 30%
    Selected from the sweep in Fig. 3 as best on AuEmoACC, EmoACC, WER, and MCD; the paper does not state whether selection used a validation split rather than the test set.
  • AuEmo classifier guidance activation threshold r = 0.7
    Set by hand in Eq. 18 with no sensitivity analysis; guidance is activated only when t > 0.7.
axioms (6)
  • domain assumption Seven basic emotion axes are sufficient to represent authentic emotional variation.
    AuEmoCodec reconstructs perceived scores on exactly seven emotion axes (§4.4) even though the paper argues that seven categories are too limited.
  • domain assumption Gemini-2.5-Flash perceived scores are valid ground truth for authentic emotion.
    No human validation or independent emotion benchmark is reported for the judge model's scores (§4.4).
  • domain assumption Training AuEmoCodec on NCSSD-EmCap provides enough emotional diversity to learn an authentic emotion token space.
    The paper trains AuEmoCodec on the same dataset as the CSS system (§5.1) and does not identify a separate large-scale emotional speech corpus.
  • domain assumption Cosine similarity weighted by distance to an AuEmo anchor is sufficient to merge redundant tokens without losing emotion information.
    AuEmoToMe (§4.2) assumes this; no information-loss analysis is reported.
  • ad hoc to paper AuEmoCodec's token assignment is a valid measure of emotional authenticity.
    AuEmoACC uses the trained AuEmoCodec as the evaluator (§5.3), creating a self-referential benchmark.
  • standard math Conditional flow matching formulation (Eqs. 14-19) is valid.
    Follows prior flow matching / CosyVoice2 work; standard derivation.
invented entities (1)
  • AuEmo token (learned discrete authentic-emotion code) no independent evidence
    purpose: Target and conditioning signal for emotion understanding and rendering; also defines the AuEmoACC evaluation metric.
    The token space is validated only by the same AuEmoCodec used in the model and by Gemini-2.5-Flash labels; no independent human or physiological benchmark shows the tokens correspond to authentic emotions.

pith-pipeline@v1.3.0-alltime-deepseek · 16371 in / 18960 out tokens · 187643 ms · 2026-08-03T01:53:46.511965+00:00 · methodology

0 comments
read the original abstract

Conversational Speech Synthesis (CSS) aims to synthesize speech with human-like emotional expression and contextual consistency in user-agent interactions. Existing CSS methods struggle to render authentic human emotions due to limited predefined emotion label spaces (e.g., seven emotion categories), while redundant multimodal tokens in multi-turn dialogue history interfere with context understanding. To address these issues, we propose AuEmoChat, a CSS framework for authentic emotion understanding and rendering. First, we develop AuEmoCodec, which learns a discrete authentic emotion token space from large-scale emotional speech via finite scalar quantization, enabling a more authentic emotion representation than limited basic emotion categories. Furthermore, we propose AuEmoToMe, an authentic-emotion-guided token merging algorithm that merges redundant tokens in multimodal dialogue history while preserving emotion-relevant context. We integrate it into an autoregressive text-speech model to predict the target authentic emotion token and speech tokens. Finally, we propose Authentic Emotion Flow Matching, which renders speech by jointly conditioning on merged dialogue context, target authentic emotion, and acoustic priors. Extensive experiments on the NCSSD-EmCap dataset demonstrate that AuEmoChat outperforms state-of-the-art CSS baselines and generates more expressive and authentic emotional speech. The code and speech demos will be available at: https://github.com/AI-S2-Lab/AuEmoChat.

Figures

Figures reproduced from arXiv: 2607.15755 by Aruukhan, Haizhou Li, Rui Liu, Yuan Zhao, Zhenqi Jia.

Figure 1
Figure 1. Figure 1: Previous CSS models rely on a limited emotion [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 1
Figure 1. Figure 1: Previous CSS models rely on a limited emotion [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: The left side illustrates the overall framework of the proposed AuEmoChat, which includes: Multimodal Dialogue [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Analysis results of different token merging rates [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

65 extracted references · 14 linked inside Pith

  1. [1]

    Swarup Ranjan Behera, Abhishek Dhiman, Karthik Gowda, and Aalekhya Satya Narayani. 2024. Fastast: Accelerating audio spectrogram transformer via token merging and cross-model knowledge distillation.arXiv preprint arXiv:2406.07676 (2024)

    Pith/arXiv arXiv 2024

  2. [2]

    Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichten- hofer, and Judy Hoffman. 2022. Token merging: Your vit but faster.arXiv preprint arXiv:2210.09461(2022)

    Pith/arXiv arXiv 2022

  3. [3]

    Qi Chen, Mingkui Tan, Yuankai Qi, Jiaqiu Zhou, Yuanqing Li, and Qi Wu. 2022. V2C: Visual voice cloning. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 21242–21251

    2022

  4. [4]

    Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, et al. 2024. Qwen2-audio technical report.arXiv preprint arXiv:2407.10759(2024)

    Pith/arXiv arXiv 2024

  5. [5]

    Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. 2014. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555(2014)

    Pith/arXiv arXiv 2014

  6. [6]

    Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multi- modality, long context, and next generation agentic capabilities.arXiv preprint arXiv:2507.06261(2025)

    Pith/arXiv arXiv 2025

  7. [7]

    Gaoxiang Cong, Jiadong Pan, Liang Li, Yuankai Qi, Yuxin Peng, Anton Van Den Hengel, Jian Yang, and Qingming Huang. 2025. Emodubber: Towards high quality and emotion controllable movie dubbing. InProceedings of the Computer Vision and Pattern Recognition Conference. 15863–15873

    2025

  8. [8]

    Yayue Deng, Jinlong Xue, Yukang Jia, Qifei Li, Yichen Han, Fengping Wang, Yingming Gao, Dengfeng Ke, and Ya Li. 2024. Concss: Contrastive-based Context Comprehension for Dialogue-Appropriate Prosody in Conversational Speech Synthesis. InICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 10706–10710

    2024

  9. [9]

    Ding Ding, Zeqian Ju, Yichong Leng, Songxiang Liu, Tong Liu, Zeyu Shang, Kai Shen, Wei Song, Xu Tan, Heyi Tang, et al . 2025. Kimi-audio technical report. arXiv preprint arXiv:2504.18425(2025)

    Pith/arXiv arXiv 2025

  10. [10]

    Zhihao Du, Qian Chen, Shiliang Zhang, Kai Hu, Heng Lu, Yexin Yang, Hangrui Hu, Siqi Zheng, Yue Gu, Ziyang Ma, et al. 2024. Cosyvoice: A scalable multilingual zero-shot text-to-speech synthesizer based on supervised semantic tokens.arXiv preprint arXiv:2407.05407(2024)

    Pith/arXiv arXiv 2024

  11. [11]

    Zhihao Du, Yuxuan Wang, Qian Chen, Xian Shi, Xiang Lv, Tianyu Zhao, Zhifu Gao, Yexin Yang, Changfeng Gao, Hui Wang, et al . 2024. Cosyvoice 2: Scal- able streaming speech synthesis with large language models.arXiv preprint arXiv:2412.10117(2024)

    Pith/arXiv arXiv 2024

  12. [12]

    Paul Ekman. 1992. An argument for basic emotions.Cognition & emotion6, 3-4 (1992), 169–200

    1992

  13. [13]

    Zhanzhou Feng and Shiliang Zhang. 2023. Efficient vision transformer via token merger.IEEE Transactions on Image Processing32 (2023), 4156–4169

    2023

  14. [14]

    Rong Fu, Ziming Wang, Chunlei Meng, Jiaxuan Lu, Jiekai Wu, Kangan Qian, Hao Zhang, and Simon Fong. 2026. Missing-by-Design: Certifiable Modality Deletion for Revocable Multimodal Sentiment Analysis.arXiv preprint arXiv:2602.16144 (2026)

    Pith/arXiv arXiv 2026

  15. [15]

    Philip Gage. 1994. A new algorithm for data compression.The C Users Journal 12, 2 (1994), 23–38

    1994

  16. [16]

    Haohan Guo, Shaofei Zhang, Frank K Soong, Lei He, and Lei Xie. 2021. Conversa- tional end-to-end tts for voice agents. In2021 IEEE Spoken Language Technology Workshop (SLT). IEEE, 403–409

    2021

  17. [17]

    Yifan Hu, Rui Liu, Yi Ren, Xiang Yin, and Haizhou Li. 2025. Chain-Talker: Chain Understanding and Rendering for Empathetic Conversational Speech Synthesis. InFindings of the Association for Computational Linguistics: ACL 2025, Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (Eds.). Association for Computational Linguistics, Vie...

    2025

  18. [18]

    Yifan Hu, Rui Liu, Yi Ren, Xiang Yin, and Haizhou Li. 2025. UniTalker: Conver- sational Speech-Visual Synthesis. InProceedings of the 33rd ACM International Conference on Multimedia. 10248–10257

    2025

  19. [19]

    Zhenqi Jia and Rui Liu. 2025. Intra-and inter-modal context interaction modeling for conversational speech synthesis. InICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 1–5

    2025

  20. [20]

    Zhenqi Jia, Rui Liu, Berrak Sisman, and Haizhou Li. 2025. Multimodal Fine- grained Context Interaction Graph Modeling for Conversational Speech Synthesis. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 8863–8869

    2025

  21. [21]

    Xue Jiang, Xiulian Peng, Huaying Xue, Yuan Zhang, and Yan Lu. 2023. Latent- domain predictive neural speech coding.IEEE/ACM Transactions on Audio, Speech, and Language Processing31 (2023), 2111–2123

    2023

  22. [22]

    Zeqian Ju, Yuancheng Wang, Kai Shen, Xu Tan, Detai Xin, Dongchao Yang, Yanqing Liu, Yichong Leng, Kaitao Song, Siliang Tang, et al. 2024. Naturalspeech 3: Zero-shot speech synthesis with factorized codec and diffusion models.arXiv preprint arXiv:2403.03100(2024)

    Pith/arXiv arXiv 2024

  23. [23]

    Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae. 2020. Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis.Advances in neural information processing systems33 (2020), 17022–17033

    2020

  24. [24]

    Robert Kubichek. 1993. Mel-cepstral distance measure for objective speech quality assessment. InProceedings of IEEE pacific rim conference on communications computers and signal processing, Vol. 1. IEEE, 125–128

    1993

  25. [25]

    Keon Lee, Kyumin Park, and Daeyoung Kim. 2023. Dailytalk: Spoken dialogue dataset for conversational text-to-speech. InICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 1–5

    2023

  26. [26]

    Jidong Leng and Qiang Yan. 2025. EIJL: Popularity Prediction of Social Media Advertisements Based on Multimodal Emotional Interaction and Joint Learning. DATA INTELLIGENCE7, 4 (2025), 1129–1146. doi:10.3724/2096-7004.di.2025.0066

    arXiv 2025

  27. [27]

    Jingbei Li, Yi Meng, Chenyi Li, Zhiyong Wu, Helen Meng, Chao Weng, and Dan Su. 2022. Enhancing speaking styles in conversational text-to-speech synthesis with graph-based multi-modal context modeling. InICASSP 2022-2022 IEEE In- ternational Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 7917–7921

    2022

  28. [28]

    Jingbei Li, Yi Meng, Xixin Wu, Zhiyong Wu, Jia Jia, Helen Meng, Qiao Tian, Yup- ing Wang, and Yuxuan Wang. 2022. Inferring speaking styles from multi-modal conversational context by multi-scale relational graph convolutional networks. In Proceedings of the 30th ACM International Conference on Multimedia. 5811–5820

    2022

  29. [29]

    Zheng Lian, Haiyang Sun, Licai Sun, Lan Chen, Haoyu Chen, Hao Gu, Zhuofan Wen, Shun Chen, Zhang Siyuan, Hailiang Yao, et al . 2024. Open-vocabulary multimodal emotion recognition: Dataset, metric, and benchmark. (2024)

    2024

  30. [30]

    Haohe Liu, Xuenan Xu, Yi Yuan, Mengyue Wu, Wenwu Wang, and Mark D Plumbley. 2024. Semanticodec: An ultra low bitrate semantic audio codec for general sound.IEEE Journal of Selected Topics in Signal Processing18, 8 (2024), 1448–1461

    2024

  31. [31]

    Rui Liu, Yifan Hu, Yi Ren, Xiang Yin, and Haizhou Li. 2024. Emotion rendering for conversational speech synthesis with heterogeneous graph-based context modeling. InProceedings of the AAAI Conference on Artificial Intelligence, Vol. 38. 18698–18706

    2024

  32. [32]

    Rui Liu, Yifan Hu, Yi Ren, Xiang Yin, and Haizhou Li. 2024. Generative expressive conversational speech synthesis. InProceedings of the 32nd ACM International Conference on Multimedia. 4187–4196

    2024

  33. [33]

    Rui Liu, Zhenqi Jia, Feilong Bao, and Haizhou Li. 2025. Retrieval-Augmented Dialogue Knowledge Aggregation for expressive conversational speech synthesis. Information Fusion(2025), 102948

    2025

  34. [34]

    Rui Liu, Zhenqi Jia, Jie Yang, Yifan Hu, and Haizhou Li. 2024. Emphasis Render- ing for Conversational Text-to-Speech with Multi-modal Multi-scale Context Modeling.arXiv preprint arXiv:2410.09524(2024)

    Pith/arXiv arXiv 2024

  35. [35]

    Rui Liu, Jia Zhenqi, Jie Yang, Yifan Hu, and Haizhou Li. 2026. Emphasis rendering for conversational text-to-speech with multi-modal multi-scale context modeling. Speech Communication(2026), 103353

    2026

  36. [36]

    Yan Liu, Renren Jin, Tianhao Shen, and Deyi Xiong. 2025. CMGBench: Bench- marking Chinese Metaphor Generation for Large Language Models.DATA INTELLIGENCE7, 4 (2025), 1270–1290. doi:10.3724/2096-7004.di.2025.0084

    arXiv 2025

  37. [37]

    Yitian Luo, Yu Liu, Lu Zhang, Feng Gao, and Jinguang Gu. 2025. A Survey on Quality Evaluation of Instruction Fine-tuning Datasets for Large Language Models.DATA INTELLIGENCE7, 3 (2025), 527–566. doi:10.3724/2096-7004.di. 2025.0021

    arXiv 2025

  38. [38]

    Ziyang Ma, Zhisheng Zheng, Jiaxin Ye, Jinchao Li, Zhifu Gao, Shiliang Zhang, and Xie Chen. 2024. emotion2vec: Self-supervised pre-training for speech emotion representation. InFindings of the Association for Computational Linguistics: ACL

    2024

  39. [39]

    2022.Conversational ai: Dialogue systems, conversational agents, and chatbots

    Michael McTear. 2022.Conversational ai: Dialogue systems, conversational agents, and chatbots. Springer Nature

    2022

  40. [40]

    Chunlei Meng, Jiabin Luo, Zhenglin Yan, Zhenyu Yu, Rong Fu, Zhongxue Gan, and Chun Ouyang. 2026. Tri-subspaces disentanglement for multimodal sentiment analysis. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 8791–8800

    2026

  41. [41]

    Fabian Mentzer, David Minnen, Eirikur Agustsson, and Michael Tschannen. 2023. Finite scalar quantization: Vq-vae made simple.arXiv preprint arXiv:2309.15505 (2023)

    Pith/arXiv arXiv 2023

  42. [42]

    Andrew Cameron Morris, Viktoria Maier, and Phil D Green. 2004. From WER and RIL to MER and WIL: improved evaluation measures for connected speech recognition.. InInterspeech. 2765–2768

    2004

  43. [43]

    Jinzhong Ning, Yuanyuan Sun, Bo Xu, Zhihao Yang, Ling Luo, and Hongfei Lin. 2024. Breaking the Boundaries: A Unified Framework for Chinese Named Entity Recognition Across Text and Speech. InFindings of the Association for Computational Linguistics: EMNLP 2024. 1250–1260

    2024

  44. [44]

    Se Park, Chae Kim, Hyeongseop Rha, Minsu Kim, Joanna Hong, Jeonghun Yeo, and Yong Ro. 2024. Let’s go real talk: Spoken dialogue model for face-to-face conversation. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 16334–16348

    2024

  45. [45]

    Robert Plutchik. 2001. The nature of emotions: Human emotions have deep evolutionary roots, a fact that may explain their complexity and provide tools for clinical practice.American scientist89, 4 (2001), 344–350

    2001

  46. [46]

    Zengzhuoma Ren, Liping Zhu, Xiaobing Zhao, and Ning Li. 2025. Amdo-Chinese Speech Translation Dataset.DATA INTELLIGENCE7, 3 (2025), 786–797. doi:10. 3724/2096-7004.di.2024.0024

    arXiv 2025

  47. [47]

    Katie Seaborn, Norihisa P Miyake, Peter Pennefather, and Mihoko Otake- Matsuura. 2021. Voice in human–agent interaction: A survey.ACM Computing Surveys (CSUR)54, 4 (2021), 1–43

    2021

  48. [48]

    Robert C Streijl, Stefan Winkler, and David S Hands. 2016. Mean opinion score (MOS) revisited: methods and applications, limitations and alternatives.Multi- media Systems22, 2 (2016), 213–227

    2016

  49. [49]

    Haobin Tang, Xulong Zhang, Jianzong Wang, Ning Cheng, and Jing Xiao. 2023. EmoMix: Emotion Mixing via Diffusion Models for Emotional Speech Synthesis. InInterspeech 2023. 12–16. doi:10.21437/Interspeech.2023-1317

    doi:10.21437/interspeech.2023-1317 2023

  50. [50]

    Aaron Van Den Oord, Oriol Vinyals, et al. 2017. Neural discrete representation learning.Advances in neural information processing systems30 (2017)

    2017

  51. [51]

    Jin Xu, Zhifang Guo, Hangrui Hu, Yunfei Chu, Xiong Wang, Jinzheng He, Yuxuan Wang, Xian Shi, Ting He, Xinfa Zhu, et al. 2025. Qwen3-omni technical report. arXiv preprint arXiv:2509.17765(2025)

    Pith/arXiv arXiv 2025

  52. [52]

    Jinlong Xue, Yayue Deng, Fengping Wang, Ya Li, Yingming Gao, Jianhua Tao, Jianqing Sun, and Jiaen Liang. 2023. M 2-ctts: End-to-end multi-scale multi-modal conversational text-to-speech synthesis. InICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 1–5

    2023

  53. [53]

    Yusuke Yasuda and Tomoki Toda. 2023. Analysis of Mean Opinion Scores in Subjective Evaluation of Synthetic Speech Based on Tail Probabilities. InProc. INTERSPEECH 2023. 5491–5495. doi:10.21437/Interspeech.2023-1285

    doi:10.21437/interspeech.2023-1285 2023

  54. [54]

    Runqing Zhang and Xue Zhou. 2025. AMNS: Attention-weighted selective mask and noise label suppression for text-to-image person retrieval. InICASSP 2025- 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 1–5

    2025

  55. [55]

    Xiaofeng Zhang, Fanshuo Zeng, Yihao Quan, Zheng Hui, and Jiawei Yao. 2025. Enhancing multimodal large language models complex reason via similarity computation. InProceedings of the AAAI Conference on Artificial Intelligence, Vol. 39. 10203–10211

    2025

  56. [56]

    Xin Zhang, Dong Zhang, Shimin Li, Yaqian Zhou, and Xipeng Qiu. 2023. Speech- tokenizer: Unified speech tokenizer for speech large language models.arXiv preprint arXiv:2308.16692(2023)

    Pith/arXiv arXiv 2023

  57. [57]

    Zhedong Zhang, Liang Li, Gaoxiang Cong, Haibing Yin, Yuhan Gao, Chenggang Yan, Anton van den Hengel, and Yuankai Qi. 2024. From speaker to dubber: movie dubbing with prosody and duration consistency learning. InProceedings of the 32nd ACM international conference on multimedia. 7523–7532

    2024

  58. [58]

    Kun Zhou, Berrak Sisman, Rajib Rana, Björn W Schuller, and Haizhou Li. 2022. Speech synthesis with mixed emotions.IEEE Transactions on Affective Computing 14, 4 (2022), 3120–3134

    2022

  59. [59]

    Li Zhou, Jianfeng Gao, Di Li, and Heung-Yeung Shum. 2020. The design and im- plementation of xiaoice, an empathetic social chatbot.Computational Linguistics 46, 1 (2020), 53–93

    2020

  60. [60]

    Nana Zhu, Zixian Feng, Hang Wang, Xing Gao, Xinyi Wang, and Yuanxing Liu

  61. [61]

    Hao Zou, Runqing Zhang, Jin Ding, Xue Zhou, Jianxiao Zou, and Mingzhu Cai

  62. [62]

    Hao Zou, Runqing Zhang, Xue Zhou, and Jianxiao Zou. 2025. GEA: Generation- Enhanced Alignment for Text-to-Image Person Retrieval.arXiv preprint arXiv:2511.10154(2025)

    arXiv 2025

  63. [2024]

    MM ’26, November 10–14, 2026, Rio de Janeiro, Brazil Jia et al

    15747–15760. MM ’26, November 10–14, 2026, Rio de Janeiro, Brazil Jia et al

    2026

  64. [2025]

    DATA INTELLIGENCE7, 2 (2025), 527–548

    Clarifying user Preference with Maximum Entropy Based Recommendation. DATA INTELLIGENCE7, 2 (2025), 527–548. doi:10.3724/2096-7004.di.2025.0040

    arXiv 2025

  65. [2026]

    InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Tackling Alignment Ambiguity in Person Retrieval through Conversational Attribute Mining. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 9689–9698