Pith. sign in

REVIEW 5 major objections 6 minor 64 references

AV-EmoDialog: Chat with Audio-Visual Users Leveraging Emotional Cues

T0 review · 5 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that a dialogue system which directly consumes raw speech and face-video inputs can generate responses that are more emotionally and contextually appropriate than text-based or cascaded multimodal approaches.

desk verdict A plausible end-to-end emotion-aware dialogue system, but the evaluation is too compromised to support the paper's central claims. read the letter →

arxiv 2412.17292 v1 pith:AHFLGDYH submitted 2024-12-23 cs.CV cs.AIcs.HC

Se Jin Park , Yeonju Kim , Hyeongseop Rha , Bella Godiva , Yong Man Ro This is my paper · ORCID
classification cs.CVcs.AIcs.HC
keywords audio-visualdialogueemotion-awareresponsegenerationempathicsystemsspeechemotionrecognitionfacialexpressionanalysismultimodallargelanguagemodelsend-to-endtrainingfine-graineddescriptions
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

AV-EmoDialog is a dialogue system that takes a user's raw speech and face video as input and, without first transcribing to text or reducing emotion to one of seven labels, generates a response together with a predicted emotion label. The paper argues that this end-to-end design preserves fine-grained verbal and non-verbal emotional cues that cascade pipelines lose. On the MultiDialog test set, the authors report that AV-EmoDialog achieves the highest emotion-alignment score (EmoBERT 0.30) and the highest BLEU-1, BLEU-4, and METEOR among compared models, and it wins the largest share of first-place human rankings. The paper also reports that removing the emotion label from training drops the EmoBERT score from 0.30 to 0.20, supporting the claim that explicit emotion tracking drives the gain.

What carries the argument

The load-bearing mechanism is the pair of learned encoders feeding a single frozen LLM. The speech encoder maps raw audio into features the LLM can interpret, trained with both ASR-style transcription and speech emotion recognition so one stream carries what is said and how it is said. The face encoder runs detected face crops through a frame encoder and a temporal encoder with learnable queries, producing a fixed-length video feature that captures how expressions change over time. The final dialogue stage keeps both encoders fixed and applies low-rank adaptation to the LLM, training it to output an emotion label followed by a response; the emotion label acts as a running summary of the user's state that persists across turns.

What would settle it

One concrete check would be to take the trained model and, on a held-out dialogue, replace the user's face video with a static neutral face or a different speaker's face while keeping the same audio track; if the generated emotion label and response do not change when the face clearly changes, the visual pathway is not contributing the emotion information the paper claims. A second check is to retrain with facial descriptions randomly shuffled across videos, and if the EmoBERT gain persists, the descriptions are not the operative supervisory signal.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central claim is that fine-grained emotional cues can be extracted from both speech and facial video and fused into a frozen instruction-tuned LLM, and that this direct audio-visual conditioning yields responses that are more emotionally aligned and no less contextually coherent than systems that convert speech to text first. The authors demonstrate this through a three-stage training scheme: a speech encoder is aligned to the LLM with speech-recognition and speech-emotion objectives; a face encoder with a temporal transformer is trained on emotion labels plus detailed facial descriptions; and the two encoders are frozen while the LLM is low-rank adapted to predict both an emotion and a response for each dialogue turn. They report the highest EmoBERT and top semantic scores, and ablation experiments showing that audio alone is worse than audio-visual and that emotion labels and facial descriptions each contribute to the gain.

Load-bearing premise

The training and evaluation assume that the emotion labels and facial descriptions used as supervision are accurate, so if those annotations do not reflect what is actually in the voice and face, the encoders may learn spurious cues and the reported emotion gains would not transfer.

Editorial extensions

If this is right

  • Cascade pipelines that transcribe speech and classify emotion into seven labels can be replaced by a single end-to-end audio-visual model without losing semantic quality.
  • Explicitly predicting an emotion label with each response gives the model a persistent emotional context that improves later turns.
  • Detailed, time-aware facial descriptions are an effective supervision signal, improving face-emotion recognition substantially over category labels alone.
  • Combining audio and video helps emotional alignment more than either audio alone or text-only input, while matching text-only semantic quality.
  • Removing emotion supervision during dialogue training measurably degrades emotional alignment, confirming the emotion signal is not incidental.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: if the reported gains hold outside MultiDialog, direct audio-visual conditioning is a plausible route to emotion-aware assistants that do not require ASR transcripts, which could matter for privacy and latency.
  • Beyond the paper: the supervision depends on a captioning model's ability to describe temporal facial dynamics from still frames; a stricter test would compare descriptions generated from actual video, which could either strengthen or weaken the reported face-emotion gain.
  • Beyond the paper: because the system outputs an emotion label alongside text, a natural next step is to condition speech synthesis on that label so the assistant's prosody carries the same emotion, making the loop fully spoken.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper presents AV-EmoDialog, a three-stage audio-visual dialogue system that (1) trains a Whisper-based speech encoder with ASR and speech-emotion recognition objectives, (2) trains a CLIP-ViT plus temporal-transformer face encoder using emotion labels and GPT-4-generated facial descriptions, and (3) LoRA fine-tunes Llama-3-Instruct to consume audio-visual features and output an emotion label plus a textual response. The system is evaluated on the MultiDialog dataset against LLaVA-Next, SpeechGPT, and a Qwen-Audio+Llama3 cascade, with additional ablations, GPT-4 evaluation, and a 10-participant human study. The paper claims that AV-EmoDialog outperforms existing multimodal LLMs in generating both emotionally appropriate and contextually appropriate responses.

Significance. The proposed architecture is a systematic and plausible integration of audio and visual emotion cues into dialogue generation, and the use of detailed facial descriptions as auxiliary supervision is an interesting idea that could benefit emotion-recognition training. The EmoBERT improvement (0.30 vs 0.22 for the best baseline) is a promising signal that auxiliary emotion modeling helps emotional alignment. However, the paper's semantic claims are contradicted by its own Table 1, the evaluation protocol as described does not provide a valid held-out comparison, and the human evaluation is too small to be conclusive. The central contribution is therefore not yet convincingly demonstrated, although the direction is worth pursuing.

major comments (5)
  1. [Section 5.1 / Table 1] The claim that "the semantic scores were also boosted, as evidenced by the highest scores in BLEU-1, BLEU-4, and METEOR" is contradicted by the paper's own data. AV-EmoDialog has lower BLEU-1 (0.193 vs 0.196 for Qwen-Audio+Llama3 and 0.212 for AV-EmoDialog*), lower ROUGE (0.198 vs 0.226 for Qwen-Audio+Llama3 and 0.215 for AV-EmoDialog*), and lower METEOR (0.164 vs 0.169 for AV-EmoDialog*); its perplexity (319.7) is worse than Qwen-Audio+Llama3 (241.9), SpeechGPT (285.7), and LLaVA-Next (317.4). Only BLEU-4 ties Qwen-Audio+Llama3 at 0.0307. Consequently, the abstract's "not only emotionally appropriate but also contextually appropriate" claim is not supported; the only quantitative metric where the proposed method is clearly best is EmoBERT.
  2. [Section 4.3] The paper states that all comparison methods were "finetuned ... on the testing dataset, MultiDialog." If the test split is used for fine-tuning or model selection, the evaluation is not a valid held-out comparison and the reported numbers do not demonstrate generalization. Please specify the exact data split, confirm that the proposed model's training data excludes the test split, and retune or fine-tune baselines only on the training split before re-reporting the tables. If the phrase is a typo for "training dataset," this should be stated explicitly and corrected.
  3. [Section 3.2 / Section 4.2 / Figure 3] The facial descriptions that supervise the face encoder are generated by GPT-4 from a collage of still frames rather than from the video itself, as stated in Section 4.2. Yet Section 3.2 and Figure 3 describe these descriptions as capturing "the progression of emotional expressions over time" and "temporal dynamics." A static collage cannot faithfully encode temporal ordering, and the model may learn spurious correlations from frame layout rather than genuine facial motion. Please provide evidence that the descriptions convey temporal information (e.g., human validation of description accuracy or a comparison with video-based captioning), or temper the claims about temporal dynamics.
  4. [Sections 5.2 and 5.3 / Tables 2 and 3] The GPT-4 and human evaluations are reported without confidence intervals, significance tests, or measures of inter-annotator agreement, and the human study uses only 10 participants. As a result, it is not possible to assess whether differences such as the 42.04% versus 34.39% first-place preference in Table 3 are statistically reliable. Please provide per-item scores, bootstrap confidence intervals, significance tests, and a justification of or increase in the number of participants.
  5. [Section 5.5 / Table 4] The ablation discussion says that adding detailed facial descriptions (EMR+EMD) "greatly increased" face emotion recognition and that semantic performance "remains competitive, surpassing that of the baselines." However, Table 4 shows that EMR+EMD decreases BLEU-1 (0.193 vs 0.219), ROUGE (0.198 vs 0.222), and METEOR (0.164 vs 0.166) relative to EMR alone, and PPL is not reported. Moreover, against the Qwen-Audio+Llama3 cascade, both ROUGE (0.198 vs 0.226) and BLEU-1 (0.193 vs 0.196) are lower. The trade-off is real and should be reported honestly rather than described as a semantic boost.
minor comments (6)
  1. [Section 5.5] The text "We evaluated the training schemes used for the speech and face encoders in Table 3" refers to the wrong table; the corresponding results appear in Table 4.
  2. [Table 2 and Figure 8] There are typographical errors: "Fleuncy" in Table 2 should be "Fluency," and "SpeehGPT" / "SpeecghGPT" in Figure 8 and Section 12 should be "SpeechGPT."
  3. [Sections 3.1 and 4.2] The speech encoder is described as a Whisper model in Section 3.1 but as a "pre-trained Whisper model of Qwen-Audio" in Section 4.2; please clarify the exact architecture and whether ASR/SER are trained through the frozen LLM or through a separate head.
  4. [Equation (1)] The notation "eai r . T ai r" appears to contain a period where a separator such as a comma is intended; please define each symbol and the superscript/subscript conventions explicitly.
  5. [Section 4.1] The perplexity evaluation cites reference [10] (Bengio et al.), but a more standard citation for dialogue-response perplexity evaluation, such as the DialogGPT paper, would be more appropriate.
  6. [Supplementary Section 11] The supplementary material states "We will open-source the annotations" but provides no link or timeline for the code, models, or annotations; please include an availability statement if the paper is to be considered reproducible.

Circularity Check

0 steps flagged · score 2.0 of 10

No construction-level circularity: the training objectives (ASR/SER, facial-description prediction, dialogue language modeling) are not identical to the evaluation metrics (BLEU/ROUGE/METEOR/PPL, EmoBERT, human/GPT rankings).

full rationale

The derivation chain is: (1) the speech encoder is trained with ASR and speech-emotion-recognition objectives on GigaSpeech, CommonVoice, LibriSpeech, RAVDESS, CREMA-D, and MultiDialog; (2) the face encoder is trained to predict emotion labels and GPT-4-generated facial descriptions; (3) the LLM is LoRA-fine-tuned with Eq. (1) to predict the AI emotion and text response from the user's audio and visual features. Evaluation uses BLEU, ROUGE, METEOR, PPL, DISTINCT, EmoBERT (a BERT model fine-tuned on GoEmotions), GPT-4 ratings, and human rankings. None of these metrics is the training loss itself, and no equation in the paper reduces the reported result to its own input. The notable same-author citation is MultiDialog [37], which is used as the dialogue corpus for training and evaluation; this is a self-citation, but MultiDialog is a collected dataset rather than an unverified theorem, so the model's outputs are not forced by construction. The paper's statement that 'the semantic scores were also boosted, as evidenced by the highest scores in BLEU-1, BLEU-4, and METEOR' is contradicted by Table 1 (AV-EmoDialog has BLEU-1 0.193 vs 0.212 for AV-EmoDialog* and 0.196 for Qwen-Audio+Llama3, and METEOR 0.164 vs 0.169 for AV-EmoDialog*), and finetuning baselines 'on the testing dataset' is a protocol flaw. These are correctness and validity concerns, not circularity. Overall, no specific circular step can be exhibited, so the circularity score is low.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the quality of the MultiDialog dataset and its emotion annotations, the reliability of GPT-4-generated facial descriptions, and the integrity of the evaluation protocol. No mathematical axioms are involved; the burdens are empirical assumptions.

free parameters (3)
  • LoRA rank = 16
    Chosen hyperparameter for LLM adaptation; no search reported, and the final performance likely depends on it.
  • Learnable query count = 128
    Number of learnable queries in the temporal encoder for video feature aggregation; set by the authors, not justified by experiments.
  • Number of temporal encoder layers = 6
    Architecture choice for the face temporal encoder; no ablation provided.
assumptions (4)
  • domain assumption MultiDialog emotion annotations are accurate and consistent
    The final-stage training and evaluation rely on these labels to teach the model to predict emotion and to score EmoBERT; the paper does not verify inter-annotator agreement.
  • domain assumption GPT-4-generated facial descriptions are reliable supervision
    Section 3.2 uses GPT-4 descriptions of face video collages as fine-grained targets; any hallucination or bias in these descriptions is propagated to the face encoder.
  • domain assumption Whisper features preserve linguistic content after fine-tuning with speech emotion recognition
    The speech encoder is fine-tuned on ASR and SER objectives; the paper assumes this does not degrade the transcription ability that downstream dialogue relies on.
  • ad hoc to paper The test split of MultiDialog is not used for training any model
    Section 4.3 states baselines were fine-tuned on the 'testing dataset'; if taken literally, this violates the assumption and invalidates the comparison.

how reviews work

0 comments
Cite this review

Pith. "Pith review of AV-EmoDialog: Chat with Audio-Visual Users Leveraging Emotional Cues." pith.science (2026). https://pith.science/paper/AHFLGDYH

@misc{pith2026241217292,
  author       = {Pith},
  title        = {Pith review of: AV-EmoDialog: Chat with Audio-Visual Users Leveraging Emotional Cues},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AHFLGDYH}},
  note         = {Machine review of arXiv:2412.17292}
}
read the original abstract

In human communication, both verbal and non-verbal cues play a crucial role in conveying emotions, intentions, and meaning beyond words alone. These non-linguistic information, such as facial expressions, eye contact, voice tone, and pitch, are fundamental elements of effective interactions, enriching conversations by adding emotional and contextual depth. Recognizing the importance of non-linguistic content in communication, we present AV-EmoDialog, a dialogue system designed to exploit verbal and non-verbal information from users' audio-visual inputs to generate more responsive and empathetic interactions. AV-EmoDialog systematically exploits the emotional cues in audio-visual dialogues; extracting speech content and emotional tones from speech, analyzing fine-grained facial expressions from visuals, and integrating these cues to generate emotionally aware responses in an end-to-end manner. Through extensive experiments, we validate that the proposed AV-EmoDialog outperforms existing multimodal LLMs in generating not only emotionally appropriate but also contextually appropriate responses.

Figures

Figures reproduced from arXiv: 2412.17292 by the authors.

Figure 1
Figure 1. Overview of the proposed method where (a) the speech encoder is trained to extract verbal cues that can be understood by the [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 3
Figure 3. Examples of emotion-relevant descriptions from face [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 2
Figure 2. Hard prompts given at each stage of training. (a) speech [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Generation results from our AV-EmoDialog, which pro [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Facial descriptions generated for training using GPT. [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]
Figure 6
Figure 6. Figure 6: GPT Evaluation prompt to evaluate the response in terms of emotional context, empathy, and fluency, . [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]
Figure 7
Figure 7. Figure 7: Human evaluation guideline for participants. [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

64 extracted references · 27 canonical work pages

  1. [1]

    Empathy through mul- timodality in conversational interfaces

    Mahyar Abbasian, Iman Azimi, Mohammad Feli, Amir M Rahmani, and Ramesh Jain. Empathy through mul- timodality in conversational interfaces. arXiv preprint arXiv:2405.04777, 2024. 1

    arXiv 2024

  2. [2]

    Rahmani, and Ramesh Jain

    Mahyar Abbasian, Iman Azimi, Mohammad Feli, Amir M. Rahmani, and Ramesh Jain. Empathy through multimodality in conversational interfaces, 2024. 2

    work page 2024

  3. [3]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ah- mad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 ,

    arXiv

  4. [4]

    Facechat: An emotion-aware face-to-face dialogue framework

    Deema Alnuhait, Qingyang Wu, and Zhou Yu. Facechat: An emotion-aware face-to-face dialogue framework. arxiv

    work page

  5. [5]

    Palm 2 technical report

    Rohan Anil, Andrew M Dai, Orhan Firat, Melvin John- son, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm 2 technical report. arXiv preprint arXiv:2305.10403, 2023. 1

    arXiv 2023

  6. [6]

    Com- mon voice: A massively-multilingual speech corpus

    Rosana Ardila, Megan Branson, Kelly Davis, Michael Hen- retty, Michael Kohler, Josh Meyer, Reuben Morais, Lind- say Saunders, Francis M Tyers, and Gregor Weber. Com- mon voice: A massively-multilingual speech corpus. arXiv preprint arXiv:1912.06670, 2019. 3, 4, 1

    arXiv 1912

  7. [7]

    Non-verbal communication in human social interaction

    Michael Argyle. Non-verbal communication in human social interaction. Non-verbal communication, 2(1), 1972. 1

    work page 1972

  8. [8]

    Qwen technical report

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023. 2

    arXiv 2023

Show all 64 references
  1. [9]

    Meteor: An automatic metric for mt evaluation with improved correlation with hu- man judgments

    Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with hu- man judgments. In Proceedings of the acl workshop on in- trinsic and extrinsic evaluation measures for machine trans- lation and/or summarization, pages 65–72, 2...

    2005

  2. [10]

    A neural probabilistic language model

    Yoshua Bengio, Réjean Ducharme, and Pascal Vincent. A neural probabilistic language model. Advances in neural in- formation processing systems, 13, 2000. 4, 1

    2000

  3. [11]

    Lan- guage models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Sub- biah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakan- tan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Lan- guage models are few-shot learners. Advances in neural in- formation processing systems, 33:1877–1901, 2020. 1

    1901

  4. [12]

    How far are we from solving the 2d & 3d face alignment problem? (and a dataset of 230,000 3d facial landmarks)

    Adrian Bulat and Georgios Tzimiropoulos. How far are we from solving the 2d & 3d face alignment problem? (and a dataset of 230,000 3d facial landmarks). In International Conference on Computer Vision, 2017. 4

    2017

  5. [13]

    Crema-d: Crowd-sourced emotional multimodal actors dataset

    Houwei Cao, David G Cooper, Michael K Keutmann, Ruben C Gur, Ani Nenkova, and Ragini Verma. Crema-d: Crowd-sourced emotional multimodal actors dataset. IEEE transactions on affective computing, 5(4):377–390, 2014. 3, 4, 1

    2014

  6. [14]

    End-to- end object detection with transformers

    Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to- end object detection with transformers. In European confer- ence on computer vision, pages 213–229. Springer, 2020. 3

    2020

  7. [15]

    Gigaspeech: An evolving, multi-domain asr corpus with 10,000 hours of transcribed audio

    Guoguo Chen, Shuzhou Chai, Guanbo Wang, Jiayu Du, Wei- Qiang Zhang, Chao Weng, Dan Su, Daniel Povey, Jan Trmal, Junbo Zhang, et al. Gigaspeech: An evolving, multi-domain asr corpus with 10,000 hours of transcribed audio. arXiv preprint arXiv:2106.06909, 2021. 3, 4, 1

    2021 arXiv

  8. [16]

    Palm: Scaling language modeling with pathways

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240): 1–113, 2023. 2

    2023

  9. [17]

    Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models

    Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shil- iang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou. Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models. arXiv preprint arXiv:2311.07919, 2023. 2, 3, 4, 5, 6

    2023 arXiv

  10. [18]

    Towards multimodal emotional support con- versation systems, 2024

    Yuqi Chu, Lizi Liao, Zhiyuan Zhou, Chong-Wah Ngo, and Richang Hong. Towards multimodal emotional support con- versation systems, 2024. 1, 2

    2024

  11. [19]

    Goemo- tions: A dataset of fine-grained emotions

    Dorottya Demszky, Dana Movshovitz-Attias, Jeongwoo Ko, Alan Cowen, Gaurav Nemade, and Sujith Ravi. Goemo- tions: A dataset of fine-grained emotions. arXiv preprint arXiv:2005.00547, 2020. 4, 1

    2005 arXiv

  12. [20]

    Retinaface: Single-shot multi- level face localisation in the wild

    Jiankang Deng, Jia Guo, Evangelos Ververas, Irene Kot- sia, and Stefanos Zafeiriou. Retinaface: Single-shot multi- level face localisation in the wild. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5203–5212, 2020. 4

    2020

  13. [21]

    Empathyear: An open-source avatar multi- modal empathetic chatbot

    Hao Fei, Han Zhang, Bin Wang, Lizi Liao, Qian Liu, and Erik Cambria. Empathyear: An open-source avatar multi- modal empathetic chatbot. arXiv preprint arXiv:2406.15177,

    arXiv

  14. [22]

    Imagebind: One embedding space to bind them all,

    Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all,

  15. [23]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021. 4, 5

    2021 arXiv

  16. [24]

    Audiogpt: Understanding and generating speech, music, sound, and talking head

    Rongjie Huang, Mingze Li, Dongchao Yang, Jiatong Shi, Xuankai Chang, Zhenhui Ye, Yuning Wu, Zhiqing Hong, Ji- awei Huang, Jinglin Liu, et al. Audiogpt: Understanding and generating speech, music, sound, and talking head. arXiv preprint arXiv:2304.12995, 2023. 2

    2023 arXiv

  17. [25]

    Perceiver: General perception with iterative attention

    Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira. Perceiver: General perception with iterative attention. In International confer- ence on machine learning, pages 4651–4664. PMLR, 2021. 3

    2021

  18. [26]

    Collavo: Crayon large language and vision model

    Byung-Kwan Lee, Beomchan Park, Chae Won Kim, and Yong Man Ro. Collavo: Crayon large language and vision model. arXiv preprint arXiv:2402.11248, 2024. 2

    2024 arXiv

  19. [27]

    A diversity-promoting objective func- tion for neural conversation models

    Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. A diversity-promoting objective func- tion for neural conversation models. arXiv preprint arXiv:1510.03055, 2015. 4

    2015 arXiv

  20. [28]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In In- ternational conference on machine learning , pages 19730– 19742. PMLR, 2023. 2

    2023

  21. [29]

    Rouge: A package for automatic evaluation of summaries

    Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out , pages 74–81, 2004. 4, 1

    2004

  22. [30]

    Paralinguistics-enhanced large language modeling of spoken dialogue

    Guan-Ting Lin, Prashanth Gurunath Shivakumar, Ankur Gandhe, Chao-Han Huck Yang, Yile Gu, Shalini Ghosh, Andreas Stolcke, Hung-yi Lee, and Ivan Bulyko. Paralinguistics-enhanced large language modeling of spoken dialogue. In ICASSP 2024-2024 IEEE International Conference on Acou...

    2024

  23. [31]

    Llava-next: Im- proved reasoning, ocr, and world knowledge, 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Im- proved reasoning, ocr, and world knowledge, 2024. 2, 5, 6

    2024

  24. [32]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024. 2

    2024

  25. [33]

    The ryer- son audio-visual database of emotional speech and song (ravdess): A dynamic, multimodal set of facial and vocal expressions in north american english

    Steven R Livingstone and Frank A Russo. The ryer- son audio-visual database of emotional speech and song (ravdess): A dynamic, multimodal set of facial and vocal expressions in north american english. PloS one , 13(5): e0196391, 2018. 3, 4, 1

    2018

  26. [34]

    Video-chatgpt: Towards detailed video understanding via large vision and language models

    Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fa- had Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. arXiv preprint arXiv:2306.05424, 2023. 2

    2023 arXiv

  27. [35]

    Gen- erative spoken dialogue language modeling

    Tu Anh Nguyen, Eugene Kharitonov, Jade Copet, Yossi Adi, Wei-Ning Hsu, Ali Elkahky, Paden Tomasello, Robin Al- gayres, Benoit Sagot, Abdelrahman Mohamed, et al. Gen- erative spoken dialogue language modeling. Transactions of the Association for Computational Linguistics, 11:250–266,

  28. [36]

    Librispeech: an asr corpus based on public do- main audio books

    Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an asr corpus based on public do- main audio books. In 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 5206–5210. IEEE, 2015. 3, 4, 1

    2015

  29. [37]

    Let’s go real talk: Spoken dialogue model for face-to-face conversa- tion

    Se Jin Park, Chae Won Kim, Hyeongseop Rha, Minsu Kim, Joanna Hong, Jeong Hun Yeo, and Yong Man Ro. Let’s go real talk: Spoken dialogue model for face-to-face conversa- tion. arXiv preprint arXiv:2406.07867, 2024. 3, 4, 5, 1

    2024 arXiv

  30. [38]

    A call for clarity in reporting bleu scores

    Matt Post. A call for clarity in reporting bleu scores. arXiv preprint arXiv:1804.08771, 2018. 4, 1

    2018 arXiv

  31. [39]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  32. [40]

    Robust speech recognition via large-scale weak supervision

    Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In Inter- national Conference on Machine Learning , pages 28492– 28518. PMLR, 2023. 3, 4

    2023

  33. [41]

    Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face

    Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face. Advances in Neural Information Processing Systems, 36, 2024. 2

    2024

  34. [42]

    Deepface: Closing the gap to human-level per- formance in face verification

    Yaniv Taigman, Ming Yang, Marc’Aurelio Ranzato, and Lior Wolf. Deepface: Closing the gap to human-level per- formance in face verification. In 2014 IEEE Conference on Computer Vision and Pattern Recognition , pages 1701– 1708, 2014. 2

    2014

  35. [43]

    Salmonn: Towards generic hearing abilities for large lan- guage models

    Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, and Chao Zhang. Salmonn: Towards generic hearing abilities for large lan- guage models. arXiv preprint arXiv:2310.13289, 2023. 3

    2023 arXiv

  36. [44]

    Gemini: a family of highly capable multimodal models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023. 2

    2023 arXiv

  37. [45]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. 1, 2

    2023 arXiv

  38. [46]

    Nonverbal cues in human–robot interaction: A communication studies perspec- tive

    Jacqueline Urakami and Katie Seaborn. Nonverbal cues in human–robot interaction: A communication studies perspec- tive. ACM Transactions on Human-Robot Interaction, 12(2): 1–21, 2023. 1

    2023

  39. [47]

    Visual chatgpt: Talking, drawing and editing with visual foundation models

    Chenfei Wu, Shengming Yin, Weizhen Qi, Xiaodong Wang, Zecheng Tang, and Nan Duan. Visual chatgpt: Talking, drawing and editing with visual foundation models. arXiv preprint arXiv:2303.04671, 2023. 2

    2023 arXiv

  40. [48]

    Next-gpt: Any-to-any multimodal llm

    Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua. Next-gpt: Any-to-any multimodal llm. arXiv preprint arXiv:2309.05519, 2023. 2, 3

    2023 arXiv

  41. [49]

    E-chat: Emotion-sensitive spoken dialogue system with large language models

    Hongfei Xue, Yuhao Liang, Bingshen Mu, Shiliang Zhang, Qian Chen, and Lei Xie. E-chat: Emotion-sensitive spoken dialogue system with large language models. arXiv preprint arXiv:2401.00475, 2023. 2

    2023 arXiv

  42. [50]

    Speechgpt: Empower- ing large language models with intrinsic cross-modal conver- sational abilities

    Dong Zhang, Shimin Li, Xin Zhang, Jun Zhan, Pengyu Wang, Yaqian Zhou, and Xipeng Qiu. Speechgpt: Empower- ing large language models with intrinsic cross-modal conver- sational abilities. arXiv preprint arXiv:2305.11000, 2023. 2, 3, 5, 6

    2023 arXiv

  43. [51]

    Video-llama: An instruction-tuned audio-visual language model for video un- derstanding

    Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video un- derstanding. arXiv preprint arXiv:2306.02858, 2023. 2

    2023 arXiv

  44. [52]

    Opt: Open pre-trained trans- former language models

    Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained trans- former language models. arXiv preprint arXiv:2205.01068,

    arXiv

  45. [53]

    Bertscore: Evaluating text genera- tion with bert

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Wein- berger, and Yoav Artzi. Bertscore: Evaluating text genera- tion with bert. arXiv preprint arXiv:1904.09675 , 2019. 4, 1

    1904 arXiv

  46. [54]

    Dialogpt: Large-scale generative pre-training for conversational response generation

    Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. Dialogpt: Large-scale generative pre-training for conversational response generation. arXiv preprint arXiv:1911.00536, 2019. 4 A V-EmoDialog: Chat with A...

    1911 arXiv

  47. [55]

    A higher BLEU score indicates a more natural and engaging dialogue model

    Evaluation Metrics BLEU [38] evaluates the fluency and adequacy of generated responses based on n-gram overlap. A higher BLEU score indicates a more natural and engaging dialogue model. We measure BLEU-1 and BLEU-4 score where BLEU-1 mea- sures the overlap at unigram and BLEU-...

  48. [56]

    Fluency evaluates the grammatical correctness, smoothness, and natural flow of the response

    GPT evaluation We ran GPT evaluation on the following criteria; coherence, fluency, and emotional intelligence. Fluency evaluates the grammatical correctness, smoothness, and natural flow of the response. The response should read as if it were spo- ken or written by a fluent l...

  49. [57]

    Human evaluation We conducted a human evaluation through Prolific5 to com- pare the quality of responses generated by various dialogue models. Ten English-speaking participants were recruited and tasked with ranking the responses generated by four different models based on pre...

  50. [58]

    It con- tains 2,615 hours of English speech from 92,325 voices with diverse genders, ages, and accents

    Datasets Common V oice[6] is a large-scale multilingual speech-text corpus collected from volunteers around the world. It con- tains 2,615 hours of English speech from 92,325 voices with diverse genders, ages, and accents. GigaSpeech [15] an English speech-text corpus from var...

  51. [59]

    The prompt given to the GPT is: These are the frames in a video

    Extracting Facial Descriptions We use GPT to annotate the face video with rich facial de- scriptions relevant to emotions. The prompt given to the GPT is: These are the frames in a video. In the video, a per- son expresses emotion through facial expressions. Describe the subtl...

  52. [60]

    Yeah, I think it’s unfair how the FD burns 6 tons of books

    More Generated Results We have included more generation results in Figure 8, which presents a comparison of the generation results from our proposed method against those from LLaV A- Next [31], SpeechGPT [50], and the Cascade model. Our A V-EmoDialog directly audio-visual as t...

  53. [61]

    Understanding the context is crucial for a fair evaluation

    Read the Dialogue History Carefully: This is the conversation that occurred before the chatbot's response. Understanding the context is crucial for a fair evaluation

  54. [62]

    Consider if the emotion expressed is suitable for the situation

    Evaluate the Chatbot’s Response: • Emotional Context: Assess how appropriate the emotion conveyed in response is within the context of the dialogue history. Consider if the emotion expressed is suitable for the situation. • Empathy: Evaluate how well the response demonstrate a...

  55. [63]

    Provide a Score for Each Criterion: • Emotional Context: [0 -5] • Empathy: [0 -5] • Fluency: [0 -5]

  56. [64]

    I've been feeling really down lately. Nothing seems to cheer me up

    Please provide only score. Don’t return justifications of the scores. Example: Dialogue History: User: "I've been feeling really down lately. Nothing seems to cheer me up." Chatbot: "I'm sorry to hear that. Is there anything in particular that's been bothering you more recentl...

    2003

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.