Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

This paper introduces AV-EMO-Reasoning, a benchmark that tests whether omni-modal LLMs can read and respond to a user's emotional state from voice and face together, and reports that current models underperform—especially at moderating extr

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 11:01 UTC pith:G3QNB57A

load-bearing objection Worth reading for the benchmark design, but the headline metrics and the visual-superiority claim aren't yet supported. the 4 major comments →

arxiv 2510.07355 v2 pith:G3QNB57A submitted 2025-10-08 cs.MM cs.SD

AV-EMO-Reasoning: Benchmarking Emotional Reasoning Capabilities in Omni-modal LLMS with Audio-visual Cues

Dingkun Zhou , Krish Patel , Ajay Kankipati , Akshaj Gupta , Zeyi Austin Li , Mohul Shukla , Vibhor Narang , Sara Kofman
show 9 more authors
Zongli Ye Grace Wang Xiaoyu Shi Tingle Li Guan-Ting Lin Kan Jen Cheng Huang-Cheng Chou Jiachen Lian Gopala Anumanchipalli
This is my paper
classification cs.MM cs.SD
keywords emotion reasoning benchmarkingaudio-visual emotion recognitionomni-modal large language modelscontinuous valence-arousal metricsemotional contagion and balancingspoken dialogue evaluationtalking-head synthetic datahuman perceptual evaluation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper aims to establish that emotional reasoning in omni-modal LLMs can be measured—and should be measured—jointly over voice and face, not from speech alone. It contributes AV-EMO-Reasoning, a benchmark built from synthetic single- and multi-turn dialogues with expressive speech and talking-head faces, plus a real-dialogue subset, scored by continuous valence/arousal alignment, categorical emotion labels, and human perceptual ratings. The authors' central empirical claim is that current omni-modal LLMs underperform on this task, especially when prosodic and facial cues conflict, and that fusing audio and visual signals improves emotion tracking over audio-only baselines. This matters because emotionally aware dialogue is a prerequisite for natural, adaptive human-AI interaction, and prior spoken-dialogue benchmarks did not test how well models integrate both cue channels over time.

Core claim

The central discovery, as the authors state it, is that when user emotion is tracked continuously in valence-arousal space from synchronized audio and video, omni-modal LLMs show measurable emotional contagion but weak emotional balancing: they mirror a user's emotional state better than they regulate or stabilize it, and their cross-turn scores degrade on multi-turn dialogues. The paper reports that visual cues reliably improve emotional coherence over audio-only baselines and that models can use audio-visual cues to produce more emotion-aware speech. The benchmark's three metric families—continuous, categorical, and perceptual—rank the tested models differently, which the authors read as e

What carries the argument

The machinery has three parts. First, an audio-visual continuous emotion recognizer (AV-CSER) trained on a dyadic video corpus outputs frame-level valence and arousal for both the user and the model's speech; this provides the reference trajectories behind the continuous scores: ECS (emotional contagion, how closely the model's emotion mirrors the user's), EBS (emotional balancing, how much the model moderates extreme user emotion), ESS (emotional stability), and ERS (their average), plus cross-turn variants. Second, a categorical audio-visual classifier trained on a face-to-face dialogue corpus provides turn-level emotion labels (neutral, happy, angry, sad). Third, a synthetic data pipeline

Load-bearing premise

The load-bearing premise is that AV-CSER's valence/arousal estimates—trained on one continuous-emotion corpus and applied to synthetic talking-head videos and LLM-generated speech—are accurate enough to serve as the user-emotion reference for every continuous metric; if those estimates are biased, the model rankings reflect evaluator error rather than emotional reasoning.

What would settle it

Have human annotators continuously rate valence and arousal on a sample of the synthetic test dialogues (or on the LLM-generated responses), then compute concordance between AV-CSER and the human ratings; if the concordance is markedly lower than the 0.553–0.688 range reported on its training corpus, or shows systematic bias on AI-generated faces, the benchmark's continuous scores and rankings are not measuring the models' emotional reasoning. A second quick check: show human raters paired speech-and-face clips from the synthetic set and test whether face emotion matches speech emotion; if mis

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If the benchmark is valid, future omni-modal LLMs can be scored specifically on emotion reasoning rather than on content quality alone, and releases can be compared on a common standard.
  • Because audio-visual fusion outperforms audio-only emotion tracking in the paper's evaluations, omni-modal LLMs that consume face video alongside speech should gain an advantage on this task.
  • The low EBS scores across models imply that emotional regulation—calming or modulating an upset user—is a distinct capability current systems lack, not just a side effect of content generation.
  • The disagreement among continuous, categorical, and perceptual metrics means any single-number leaderboard for emotional dialogue would be misleading; reporting all three families is necessary.
  • Human raters preferred the more verbose LLM responses while automatic metrics favored the dataset's affect fidelity, indicating perceived quality and emotional signal fidelity can diverge in real evaluation.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because the user-emotion reference in the synthetic set is generated by AV-CSER rather than by human annotators on the test items, the reported rankings are only as strong as that model's generalization; a human continuous-annotation pass on a sample of synthetic dialogues would be a direct validation.
  • The synthetic pipeline derives both the speech emotion and the face emotion from the same LLM-written style prompt plus a random face, so true cross-modal conflicts are likely rare; a deliberate mismatch condition (e.g., happy face over angry prosody) would stress the paper's claim that models are robust to disagreement.
  • The automatic-versus-perceptual divergence suggests a testable trade-off: tuning an omni-modal LLM to maximize ECS/EBS scores may lower human naturalness ratings, and vice versa; reporting both families on every system would reveal whether the two objectives conflict.
  • If the talking-head animations of random faces are not emotionally coherent with the synthesized speech, the video channel may be adding noise rather than signal; an ablation that swaps faces or uses real human videos would clarify what visual cues actually contribute.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes AV-EMO-Reasoning, a benchmark for evaluating emotion reasoning in omni-modal LLMs using synchronized audio-visual inputs. It constructs synthetic single- and multi-turn dialogues (GPT-4 scripts, CosyVoice emotional TTS, DreamTalk talking-head videos with randomly sampled AI-Face images) and a real-world subset (MultiDialog). The evaluation suite combines continuous metrics (ECS, EBS, ESS, ERS) computed from an AV-CSER model trained on RECOLA, categorical metrics from a MultiDialog-trained audio-visual classifier, and human perceptual ratings (ER, EN, RR). Three omni-modal LLMs (Baichuan-Omni-1.5, MiniCPM-o 2.6, Qwen 2.5-Omni-7B) are benchmarked against dataset baseline dialogues. The paper claims that visual cues reliably improve emotional coherence over audio-only baselines, that current LLMs underperform on audio-visual emotional reasoning, and that the three metric families capture complementary facets of performance.

Significance. If the benchmark can be trusted, it fills a genuinely useful gap: no existing benchmark systematically tests whether omni-modal LLMs can perceive and reason about a user's emotional state from synchronized speech and face. The design choice to report continuous, categorical, and perceptual metric families is methodologically attractive, and the reported AV-CSER fusion results on RECOLA (e.g., CCC Mean 0.688 on the subject-wise split) are promising. However, the validity of the benchmark as a reproducible standard currently rests on unvalidated emotion-recognition models serving as ground truth on synthetic out-of-distribution stimuli, and the paper's headline claim that visual cues improve LLM emotional coherence over audio-only baselines is not actually tested. These load-bearing issues, together with the ERS metric inconsistencies in Table 4, prevent acceptance in the present form.

major comments (4)
  1. [Sections 4, 5.1, 5.3] The continuous metrics ECS/EBS/ESS/ERS use the AV-CSER model trained on RECOLA (real dyadic video conferences, French speakers) as the emotion reference for both user and LLM turns in the synthetic corpus. No human annotation or validation of AV-CSER is provided for CosyVoice-synthesized speech or DreamTalk-animated AI-Face images. Table 3 itself shows substantial modality asymmetries (video-only valence CCC 0.468 vs audio-only 0.139 on the subject-wise split), so a systematic evaluator bias on synthetic prosody or face artifacts would propagate directly into every continuous metric and could change the Table 4 rankings. The same domain-shift concern applies to the categorical classifier trained on MultiDialog and applied to the synthetic stimuli. Please add a human-validation study of both evaluators on the actual test stimuli (e.g., CCC and confusion matrices on a sample), or at least
  2. [Abstract, Section 6, Table 4] The abstract claims that 'visual cues reliably improve emotional coherence over audio-only baselines.' The only modality comparisons in the paper are Tables 2 and 3, which compare the emotion-recognition models themselves, not the evaluated LLMs. Table 4 feeds each LLM both audio and video; there is no condition with audio only, video only, or audio-visual inputs for the LLMs. If the claim is about how omni-modal LLMs use visual cues, a proper LLM-level modality ablation is required. If the claim is instead about the underlying AV-CSER evaluator, the wording is misleading and should be corrected.
  3. [Section 2.1.1, Table 4] ERS is defined as averaging ECS, EBS, and ESS, yet the reported ERS values do not match this definition. For example, MiniCPM-o 2.6 Multi-Turn Real has ECS=0.648, EBS=0.501, ESS=0.688; the arithmetic mean is 0.612, but the table reports ERS=0.258. Similar discrepancies appear in most rows. The conditional clause 'If no extreme user emotions occur, ERS becomes the mean of ECS and ESS' is also not operationalized (no definition of 'extreme'). Since ERS is the headline continuous score, the exact formula and the table values must be reconciled.
  4. [Section 5.1] The synthetic corpus assumes that DreamTalk animation of a randomly selected AI-Face image produces facial expressions that are emotionally coherent with the CosyVoice-synthesized speech. No evidence is provided for this assumption; the random face may have a mismatched identity/gender or may not track the intended emotion at all. If the face and voice disagree in emotion, the benchmark is testing how LLMs handle synthetic incongruence rather than genuine naturalistic audio-visual emotional reasoning. Please validate the emotional coherence of the generated talking-head videos (e.g., human ratings of face emotion and speech emotion on a sample) and either filter or report the degree of mismatch.
minor comments (5)
  1. [Section 5.3, Table 4] Section 5.3 states that metrics for Angry and Sad are omitted due to low frequency, yet Table 4 contains Angry and Sad columns for the single-turn synthetic and real MultiDialog rows. Clarify which conditions exclude these categories and why.
  2. [Section 2.1.2, Table 4] The multi-turn rows in Table 4 presumably report the cross-turn CT-ECS/CT-EBS/CT-ESS/CT-ERS from Section 2.1.2, but the columns are labeled simply ECS, EBS, ESS, ERS. Add a legend or column prefix to make the aggregation clear.
  3. [References] Reference [37] is cited for MiniCPM-o 2.6 but points to the MiniCPM-V paper. Please update to the correct MiniCPM-o technical report.
  4. [Table 3 caption] Typo: 'subjective-wise split' should be 'subject-wise split'.
  5. [Section 6] The statement that 'human evaluation may be biased' because LLM-generated responses receive higher perceptual scores is not a valid inference; the difference could reflect actual content quality, verbosity, or annotation instructions. Please rephrase as an observed discrepancy rather than a conclusion about bias.

Circularity Check

0 steps flagged

No derivation-level circularity; metric self-citation and instrument dependence are validity risks, not tautological reductions.

full rationale

The paper's claimed derivation chain is an evaluation protocol, not a mathematical derivation. The continuous metrics (ECS/EBS/ESS/ERS) are reused from the authors' prior EMO-Reasoning work, and both the user-side and agent-side affect trajectories are produced by the same AV-CSER model trained on RECOLA, while categorical labels come from a classifier trained on MultiDialog. This is a real basis for concern about benchmark validity: AV-CSER is not validated on the synthetic GPT-4/CosyVoice/DreamTalk domain, and no human annotation of the synthetic test set is reported, so systematic evaluator bias could influence the Table 4 rankings. However, that is a measurement/domain-shift risk, not circularity by construction. The model rankings are not algebraically forced by the fitted parameters: AV-CSER was trained on an external corpus, the LLM responses are generated independently of AV-CSER, and the benchmark does not predict AV-CSER's outputs as the target quantity. The self-citation to EMO-Reasoning supplies metric definitions and context but no unverified uniqueness theorem or ansatz that the conclusions reduce to. The paper also includes external human perceptual ratings and reports RECOLA CCC as independent evidence for AV-CSER. Therefore, no exhibitable circular step is present; the residual issue is self-reliance on the authors' own evaluators, which warrants a low non-zero score rather than a circularity finding.

Axiom & Free-Parameter Ledger

2 free parameters · 4 axioms · 0 invented entities

The benchmark's measurements rest on two self-trained evaluator models (AV-CSER and the categorical classifier) plus the EMO-Reasoning metric stack and synthetic talking-head data. No new theoretical entities are postulated; the free parameters are learned model weights. The largest unacknowledged cost is that the user's 'true' emotion in the synthetic set is not annotated by humans—it is the authors' model output.

free parameters (2)
  • AV-CSER model parameters (BiLSTM regression head and fusion) = CCC Mean 0.553 (official RECOLA split) / 0.688 (with dev augmentation)
    Trained on RECOLA to map audio-video features to valence/arousal; used to compute every continuous metric for both user and model turns, so all ECS/EBS/ESS/ERS values inherit its biases.
  • Categorical audio-visual emotion classifier parameters = Weighted F1 ~0.59 average over four classes (Table 1)
    Trained on MultiDialog to label neutral/happy/angry/sad; categorical metrics depend on this classifier's output distribution for both user and LLM responses.
axioms (4)
  • domain assumption EMO-Reasoning metric definitions (ECS, EBS, ESS, ERS) are valid operationalizations of emotional reasoning.
    Section 2.1 adopts metrics from [10] without independent validation; if these metrics do not capture emotional reasoning, the benchmark measures something else.
  • domain assumption AV-CSER valence/arousal predictions are accurate enough to serve as ground-truth affect for both user and LLM turns.
    Section 4; no human-rated continuous labels are provided for the synthetic test set, so model outputs are used as the 'user emotion' reference.
  • ad hoc to paper DreamTalk animations of random AI-Face images display facial expressions that match the emotional prosody of the synthesized speech.
    Section 5.1: 'for each audio segment, a random facial image was selected from the AI-Face dataset and animated using DreamTalk'; no validation that the generated face is emotionally coherent with the audio, yet the benchmark assumes audio-visual coherence.
  • domain assumption Human Likert ratings from at least 5 annotators on 15 dialogues per system are reliable without reported inter-annotator agreement.
    Section 2.3; small sample and no ICC/kappa or significance testing are reported.

pith-pipeline@v1.3.0-alltime-deepseek · 9530 in / 11442 out tokens · 93640 ms · 2026-08-04T11:01:45.726426+00:00 · methodology

0 comments
read the original abstract

Emotions conveyed through voice and face shape engagement and context in human AI interaction. Despite rapid progress in omni modal large language models, the holistic evaluation of emotional reasoning with audiovisual cues remains limited. To address this gap, we introduce AV EMO Reasoning, a benchmark designed to systematically assess emotional reasoning abilities in large language models. The framework uses a curated audiovisual corpus comprising synthetic single turn and multi turn dialogues and a real world subset, together with emotion perception and interaction reasoning metrics, to evaluate whether models can understand user emotions and produce appropriate responses. By releasing a systematic evaluation benchmark, AV EMO Reasoning offers a reproducible standard for evaluating emotion aware dialogue and advances toward more natural, adaptive human AI interaction.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. S-MARC: Causal Streaming Reasoning for Full-Duplex Conversational Behavior Modeling

    cs.CL 2026-02 conditional novelty 6.0

    A streaming causal model predicts per-second two-level speech acts and rationale explanations, trained on 120 hours of LLM-synthesized duplex dialogue.

Reference graph

Works this paper leans on

57 extracted references · 21 linked inside Pith · cited by 1 Pith paper

  1. [1]

    Emotional communication in speech and music: The role of melodic and rhythmic contrasts,

    Lena Quinto et al., “Emotional communication in speech and music: The role of melodic and rhythmic contrasts,”Frontiers in psychology, vol. 4, pp. 184, 2013

    2013

  2. [2]

    Effects of variation in emotional tone of voice on speech perception,

    John W Mullennix et al., “Effects of variation in emotional tone of voice on speech perception,”Language and speech, vol. 45, no. 3, pp. 255–283, 2002

    2002

  3. [3]

    Analysis of emotion recognition using facial ex- pressions, speech and multimodal information,

    Carlos Busso et al., “Analysis of emotion recognition using facial ex- pressions, speech and multimodal information,” inProceedings of the 6th international conference on Multimodal interfaces, 2004, pp. 205– 211

    2004

  4. [4]

    Language models are few-shot learners,

    Tom Brown et al., “Language models are few-shot learners,”Ad- vances in neural information processing systems, vol. 33, pp. 1877– 1901, 2020

    1901

  5. [5]

    Llama-omni: Seamless speech interaction with large language models,

    Qingkai Fang et al., “Llama-omni: Seamless speech interaction with large language models,”arXiv preprint arXiv:2409.06666, 2024

    Pith/arXiv arXiv 2024

  6. [6]

    Mini-omni: Language mod- els can hear, talk while thinking in streaming,

    Zhifei Xie and Changqiao Wu, “Mini-omni: Language mod- els can hear, talk while thinking in streaming,”arXiv preprint arXiv:2408.16725, 2024

    Pith/arXiv arXiv 2024

  7. [7]

    Freeze-omni: A smart and low latency speech-to-speech dialogue model with frozen llm,

    Xiong Wang et al., “Freeze-omni: A smart and low latency speech-to-speech dialogue model with frozen llm,”arXiv preprint arXiv:2411.00774, 2024

    Pith/arXiv arXiv 2024

  8. [8]

    Glm-4-voice: Towards intelligent and human-like end-to-end spoken chatbot,

    Aohan Zeng et al., “Glm-4-voice: Towards intelligent and human-like end-to-end spoken chatbot,”arXiv preprint arXiv:2412.02612, 2024

    Pith/arXiv arXiv 2024

  9. [9]

    Moshi: a speech-text foundation model for real-time dialogue,

    Alexandre D ´efossez et al., “Moshi: a speech-text foundation model for real-time dialogue,”arXiv preprint arXiv:2410.00037, 2024

    Pith/arXiv arXiv 2024

  10. [10]

    EMO-Reasoning: Benchmarking Emotional Rea- soning Capabilities in Spoken Dialogue Systems,

    Jingwen Liu et al., “EMO-Reasoning: Benchmarking Emotional Rea- soning Capabilities in Spoken Dialogue Systems,”2025 IEEE Auto- matic Speech Recognition and Understanding Workshop (ASRU), 2025

    2025

  11. [11]

    Textually pretrained speech language mod- els,

    Michael Hassid et al., “Textually pretrained speech language mod- els,”Advances in Neural Information Processing Systems, vol. 36, pp. 63483–63501, 2023

    2023

  12. [12]

    Gsqa: An end-to-end model for generative spo- ken question answering,

    Min-Han Shih et al., “Gsqa: An end-to-end model for generative spo- ken question answering,”arXiv preprint arXiv:2312.09781, 2023

    Pith/arXiv arXiv 2023

  13. [13]

    Align-slm: Textless spoken language mod- els with reinforcement learning from ai feedback,

    Guan-Ting Lin et al., “Align-slm: Textless spoken language mod- els with reinforcement learning from ai feedback,”arXiv preprint arXiv:2411.01834, 2024

    Pith/arXiv arXiv 2024

  14. [14]

    Dynamic-superb: Towards a dynamic, collab- orative, and comprehensive instruction-tuning benchmark for speech,

    Chien-yu Huang et al., “Dynamic-superb: Towards a dynamic, collab- orative, and comprehensive instruction-tuning benchmark for speech,” inICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024, pp. 12136– 12140

    2024

  15. [15]

    Dynamic-superb phase-2: A collaboratively expanding benchmark for measuring the capabilities of spoken lan- guage models with 180 tasks,

    Chien-yu Huang et al., “Dynamic-superb phase-2: A collaboratively expanding benchmark for measuring the capabilities of spoken lan- guage models with 180 tasks,”arXiv preprint arXiv:2411.05361, 2024

    Pith/arXiv arXiv 2024

  16. [16]

    Sd-eval: A benchmark dataset for spoken dialogue understanding beyond words,

    Junyi Ao et al., “Sd-eval: A benchmark dataset for spoken dialogue understanding beyond words,”Advances in Neural Information Pro- cessing Systems, vol. 37, pp. 56898–56918, 2024

    2024

  17. [17]

    Advancing large language models to capture varied speaking styles and respond properly in spoken conversations,

    Guan-Ting Lin et al., “Advancing large language models to capture varied speaking styles and respond properly in spoken conversations,” arXiv preprint arXiv:2402.12786, 2024

    Pith/arXiv arXiv 2024

  18. [18]

    Can LLMs Understand the Im- plication of Emphasized Sentences in Dialogue?,

    Guan-Ting Lin and Hung-yi Lee, “Can LLMs Understand the Im- plication of Emphasized Sentences in Dialogue?,”arXiv preprint arXiv:2406.11065, 2024

    Pith/arXiv arXiv 2024

  19. [19]

    Paralinguistics-enhanced large language model- ing of spoken dialogue,

    Guan-Ting Lin et al., “Paralinguistics-enhanced large language model- ing of spoken dialogue,” inICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024, pp. 10316–10320

    2024

  20. [20]

    Full-duplex-bench: A benchmark to evaluate full-duplex spoken dialogue models on turn-taking capabilities,

    Guan-Ting Lin et al., “Full-duplex-bench: A benchmark to evaluate full-duplex spoken dialogue models on turn-taking capabilities,”arXiv preprint arXiv:2503.04721, 2025

    Pith/arXiv arXiv 2025

  21. [21]

    Full-Duplex-Bench v1. 5: Evaluating Over- lap Handling for Full-Duplex Speech Models,

    Guan-Ting Lin et al., “Full-Duplex-Bench v1. 5: Evaluating Over- lap Handling for Full-Duplex Speech Models,”arXiv preprint arXiv:2507.23159, 2025

    Pith/arXiv arXiv 2025

  22. [22]

    Emonet-face: An expert-annotated benchmark for synthetic emotion recognition,

    Christoph Schuhmann et al., “Emonet-face: An expert-annotated benchmark for synthetic emotion recognition,”arXiv preprint arXiv:2505.20033, 2025

    Pith/arXiv arXiv 2025

  23. [23]

    Dfme: A new benchmark for dynamic facial micro- expression recognition,

    Sirui Zhao et al., “Dfme: A new benchmark for dynamic facial micro- expression recognition,”IEEE Transactions on Affective Computing, vol. 15, no. 3, pp. 1371–1386, 2023

    2023

  24. [24]

    What comprises a good talking-head video gener- ation?: A survey and benchmark,

    Lele Chen et al., “What comprises a good talking-head video gener- ation?: A survey and benchmark,”arXiv preprint arXiv:2005.03201, 2020

    Pith/arXiv arXiv 2005

  25. [25]

    Subjective and objective quality-of-experience assessment for 3d talking heads,

    Yingjie Zhou et al., “Subjective and objective quality-of-experience assessment for 3d talking heads,” inProceedings of the 32nd ACM International Conference on Multimedia, 2024, pp. 6033–6042

    2024

  26. [26]

    Av- data2vec: Self-supervised learning of audio-visual speech representa- tions with contextualized target representations,

    Jiachen Lian, Alexei Baevski, Wei-Ning Hsu, and Michael Auli, “Av- data2vec: Self-supervised learning of audio-visual speech representa- tions with contextualized target representations,” in2023 IEEE Au- tomatic Speech Recognition and Understanding Workshop (ASRU). IEEE, 2023, pp. 1–8

    2023

  27. [27]

    Jointly Learning From Unimodal and Multimodal-Rated Labels in Audio-Visual Emotion Recognition,

    Lucas Goncalves et al., “Jointly Learning From Unimodal and Multimodal-Rated Labels in Audio-Visual Emotion Recognition,” IEEE Open Journal of Signal Processing, vol. 6, pp. 165–174, 2025

    2025

  28. [28]

    Stimulus Modality Matters: Impact of Perceptual Evaluations from Different Modalities on Speech Emotion Recognition System Performance,

    Huang-Cheng Chou et al., “Stimulus Modality Matters: Impact of Perceptual Evaluations from Different Modalities on Speech Emotion Recognition System Performance,” inICASSP 2025 - 2025 IEEE In- ternational Conference on Acoustics, Speech and Signal Processing (ICASSP), 2025, pp. 1–5

    2025

  29. [29]

    Cross-modal incongruity aligning and collaborating for multi-modal sarcasm detection,

    Jie Wang et al., “Cross-modal incongruity aligning and collaborating for multi-modal sarcasm detection,”Information Fusion, vol. 103, pp. 102132, 2024

    2024

  30. [30]

    Let’s Go Real Talk: Spoken Dialogue Model for Face-to-Face Conversation,

    Se Jin Park et al., “Let’s Go Real Talk: Spoken Dialogue Model for Face-to-Face Conversation,”Association for Computational Linguis- tics (ACL), 2024

    2024

  31. [31]

    FunAudioLLM: V oice Understanding and Genera- tion Foundation Models for Natural Interaction Between Humans and LLMs,

    Keyu An, , et al., “FunAudioLLM: V oice Understanding and Genera- tion Foundation Models for Natural Interaction Between Humans and LLMs,”arXiv preprint arXiv:2407.04051, 2024

    Pith/arXiv arXiv 2024

  32. [32]

    Facial Expression Recognition with Adaptive Frame Rate based on Multiple Testing Correction,

    Andrey Savchenko, “Facial Expression Recognition with Adaptive Frame Rate based on Multiple Testing Correction,”International Con- ference on Machine Learning (ICML), 2023

    2023

  33. [33]

    Introducing the RECOLA multimodal corpus of remote collaborative and affective interactions,

    Fabien Ringeval et al., “Introducing the RECOLA multimodal corpus of remote collaborative and affective interactions,” in2013 10th IEEE international conference and workshops on automatic face and gesture recognition (FG). IEEE, 2013, pp. 1–8

    2013

  34. [34]

    Avec 2016: Depression, mood, and emotion recognition workshop and challenge,

    Michel Valstar et al., “Avec 2016: Depression, mood, and emotion recognition workshop and challenge,” inProceedings of the 6th inter- national workshop on audio/visual emotion challenge, 2016, pp. 3–10

    2016

  35. [35]

    Hicmae: Hierarchical contrastive masked autoencoder for self-supervised audio-visual emotion recognition,

    Licai Sun et al., “Hicmae: Hierarchical contrastive masked autoencoder for self-supervised audio-visual emotion recognition,”Information Fu- sion, vol. 108, pp. 102382, 2024

    2024

  36. [36]

    Baichuan-Omni-1.5 Technical Report,

    Yadong Li, , et al., “Baichuan-Omni-1.5 Technical Report,”arXiv preprint arXiv:2501.15368, 2025

    arXiv 2025

  37. [37]

    MiniCPM-V: A GPT-4V Level MLLM on Your Phone,

    Yuan Yao et al., “MiniCPM-V: A GPT-4V Level MLLM on Your Phone,”arXiv preprint arXiv:2408.01800, 2024

    Pith/arXiv arXiv 2024

  38. [38]

    Qwen2.5-Omni Technical Report,

    Jin Xu et al., “Qwen2.5-Omni Technical Report,”arXiv preprint arXiv:2503.20215, 2025

    Pith/arXiv arXiv 2025

  39. [39]

    Gpt-4 technical report,

    Josh Achiam et al., “Gpt-4 technical report,”arXiv preprint arXiv:2303.08774, 2023

    Pith/arXiv arXiv 2023

  40. [40]

    Cosyvoice 2: Scalable streaming speech synthesis with large language models,

    Zhihao Du et al., “Cosyvoice 2: Scalable streaming speech synthesis with large language models,”arXiv preprint arXiv:2412.10117, 2024

    Pith/arXiv arXiv 2024

  41. [41]

    AI-Face: A Million-Scale Demographically Annotated AI-Generated Face Dataset and Fairness Benchmark,

    Li Lin et al., “AI-Face: A Million-Scale Demographically Annotated AI-Generated Face Dataset and Fairness Benchmark,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition (CVPR), 2025

    2025

  42. [42]

    DreamTalk: When Expressive Talking Head Generation Meets Diffusion Probabilistic Models,

    Yifeng Ma et al., “DreamTalk: When Expressive Talking Head Generation Meets Diffusion Probabilistic Models,”arXiv preprint arXiv:2312.09767, 2023

    Pith/arXiv arXiv 2023

  43. [43]

    Unconstrained dysfluency modeling for dys- fluent speech transcription and detection,

    Jiachen Lian, Carly Feng, Naasir Farooqi, Steve Li, Anshul Kashyap, Cheol Jun Cho, Peter Wu, Robbie Netzorg, Tingle Li, and Gopala Kr- ishna Anumanchipalli, “Unconstrained dysfluency modeling for dys- fluent speech transcription and detection,” in2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). IEEE, 2023, pp. 1–8

    2023

  44. [44]

    Towards hierarchical spo- ken language disfluency modeling,

    Jiachen Lian and Gopala Anumanchipalli, “Towards hierarchical spo- ken language disfluency modeling,” inProceedings of the 18th Con- ference of the European Chapter of the Association for Computational Linguistics (V olume 1: Long Papers), 2024

    2024

  45. [45]

    Ssdm: Scalable speech dysfluency model- ing,

    Jiachen Lian, Xuanru Zhou, Zoe Ezzes, Jet V onk, Brittany Morin, David Paul Baquirin, Zachary Miller, Maria Luisa Gorno Tempini, and Gopala Anumanchipalli, “Ssdm: Scalable speech dysfluency model- ing,” inAdvances in Neural Information Processing Systems, 2024, vol. 37

    2024

  46. [46]

    Auto- matic detection of articulatory-based disfluencies in primary progres- sive aphasia,

    Jiachen Lian, Xuanru Zhou, Chenxu Guo, Zongli Ye, Zoe Ezzes, Jet V onk, Brittany Morin, David Baquirin, Zachary Mille, Maria Luisa Gorno Tempini, and Gopala Krishna Anumanchipalli, “Auto- matic detection of articulatory-based disfluencies in primary progres- sive aphasia,”IEEE JSTSP, 2025

    2025

  47. [47]

    Yolo-stutter: End-to-end region-wise speech dysfluency detection,

    Xuanru Zhou, Anshul Kashyap, Steve Li, Ayati Sharma, Brittany Morin, David Baquirin, Jet V onk, Zoe Ezzes, Zachary Miller, Maria Tempini, Jiachen Lian, and Gopala Anumanchipalli, “Yolo-stutter: End-to-end region-wise speech dysfluency detection,” inInterspeech 2024, 2024, pp. 937–941

    2024

  48. [48]

    Stutter-solver: End-to-end multi- lingual dysfluency detection,

    Xuanru Zhou, Cheol Jun Cho, Ayati Sharma, Brittany Morin, David Baquirin, Jet V onk, Zoe Ezzes, Zachary Miller, Boon Lead Tee, Maria Luisa Gorno-Tempini, et al., “Stutter-solver: End-to-end multi- lingual dysfluency detection,” in2024 IEEE Spoken Language Tech- nology Workshop (SLT). IEEE, 2024, pp. 1039–1046

    2024

  49. [49]

    Time and tokens: Benchmarking end-to-end speech dys- fluency detection,

    Xuanru Zhou, Jiachen Lian, Cheol Jun Cho, Jingwen Liu, Zongli Ye, Jinming Zhang, Brittany Morin, David Baquirin, Jet V onk, Zoe Ezzes, Zachary Miller, Maria Luisa Gorno Tempini, and Gopala Anu- manchipalli, “Time and tokens: Benchmarking end-to-end speech dys- fluency detection,” 2024

    2024

  50. [50]

    Towards accurate phonetic error detection through phoneme similarity modeling,

    Xuanru Zhou, Jiachen Lian, Cheol Jun Cho, Tejas Prabhune, Shuhe Li, William Li, Rodrigo Ortiz, Zoe Ezzes, Jet V onk, Brittany Morin, Rian Bogley, Lisa Wauters, Zachary Miller, Maria Gorno-Tempini, and Gopala Anumanchipalli, “Towards accurate phonetic error detection through phoneme similarity modeling,”Interspeech, 2025

    2025

  51. [51]

    Dysfluent wfst: A framework for zero-shot speech dysfluency transcription and detection,

    Chenxu Guo, Jiachen Lian, Xuanru Zhou, Jinming Zhang, Shuhe Li, Zongli Ye, Hwi Joo Park, Anaisha Das, Zoe Ezzes, Jet V onk, Brit- tany Morin, Rian Bogley, Lisa Wauters, Zachary Miller, Maria Gorno- Tempini, and Gopala Anumanchipalli, “Dysfluent wfst: A framework for zero-shot speech dysfluency transcription and detection,”Inter- speech, 2025

    2025

  52. [52]

    Analysis and evaluation of synthetic data generation in speech dysfluency detec- tion,

    Jinming Zhang, Xuanru Zhou, Jiachen Lian, Shuhe Li, William Li, Zoe Ezzes, Rian Bogley, Lisa Wauters, Zachary Miller, Jet V onk, Brittany Morin, Maria Gorno-Tempini, and Gopala Anumanchipalli, “Analysis and evaluation of synthetic data generation in speech dysfluency detec- tion,”Interspeech, 2025

    2025

  53. [53]

    Lcs-ctc: Leveraging soft alignments to enhance phonetic transcription robustness,

    Zongli Ye, Jiachen Lian, Akshaj Gupta, Xuanru Zhou, Haodong Li, Kr- ish Patel, Hwi Joo Park, Dingkun Zhou, Chenxu Guo, Shuhe Li, et al., “Lcs-ctc: Leveraging soft alignments to enhance phonetic transcription robustness,”arXiv preprint arXiv:2508.03937, 2025

    Pith/arXiv arXiv 2025

  54. [54]

    Seamless dysfluent speech text alignment for disordered speech analysis,

    Zongli Ye, Jiachen Lian, Xuanru Zhou, Jinming Zhang, Haodong Li, Shuhe Li, Chenxu Guo, Anaisha Das, Peter Park, Zoe Ezzes, Jet V onk, Brittany Morin, Rian Bogley, Lisa Wauters, Zachary Miller, Maria Gorno-Tempini, and Gopala Anumanchipalli, “Seamless dysfluent speech text alignment for disordered speech analysis,”Interspeech, 2025

    2025

  55. [55]

    K-function: Joint pronunciation transcription and feedback for evaluating kids language function,

    Shuhe Li, Chenxu Guo, Jiachen Lian, Cheol Jun Cho, Wenshuo Zhao, Xuanru Zhou, Dingkun Zhou, Sam Wang, Grace Wang, Jingze Yang, et al., “K-function: Joint pronunciation transcription and feedback for evaluating kids language function,”arXiv preprint arXiv:2507.03043, 2025

    arXiv 2025

  56. [56]

    Deep Neural Convolutive Matrix Factorization for Articulatory Representation Decomposition,

    Jiachen Lian, Alan W Black, Louis Goldstein, and Gopala Krishna Anumanchipalli, “Deep Neural Convolutive Matrix Factorization for Articulatory Representation Decomposition,” inProc. Interspeech 2022, 2022, pp. 4686–4690

    2022

  57. [57]

    Articulatory representation learn- ing via joint factor analysis and neural matrix factorization,

    Jiachen Lian, Alan W Black, Yijing Lu, Louis Goldstein, Shinji Watan- abe, and Gopala K Anumanchipalli, “Articulatory representation learn- ing via joint factor analysis and neural matrix factorization,” inICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023, pp. 1–5

    2023