Pith. sign in

REVIEW 3 major objections 6 minor 55 references

Full-face video cues make emotion-aware speech generation measurably better than audio alone.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

An audio-visual language model that adds full-face visual features to a pre-trained expressive speech model improves emotion recognition and expressive speech generation by a few F1 points over speech-only on synthetic IEMOCAP data.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection The AVSR and emotion-recognition results are solid, but the headline generation claim is confounded by each model using its own predicted emotion label; the fix is a controlled comparison. the 3 major comments →

arxiv 2508.16188 v2 pith:VH5RUQQW submitted 2025-08-22 cs.CL cs.CVcs.MMcs.SDeess.AS

Seeing is Believing: Emotion-Aware Audio-Visual Language Modeling for Expressive Speech Generation

Weiting Tan , Jiachen Lian , Hirofumi Inaguma , Paden Tomasello , Philipp Koehn , Xutai Ma This is my paper
classification cs.CL cs.CVcs.MMcs.SDeess.AS
keywords audio-visual language modelexpressive speech generationemotion recognitionmultimodal fusionQ-Formerfacial expression featuresspeech language modelIEMOCAP
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that a speech language model can use full-face visual cues to become more emotionally accurate, both at recognizing emotion and at generating spoken responses that carry the intended emotion. It builds on a pre-trained expressive speech model, injects face-derived features through a learned fusion module during pre-training, then fine-tunes on emotion recognition and expressive dialogue. Across both tasks, the audio-visual model beats the same model without vision, with the central reported gains being +4.9 macro F1 in emotion recognition and +4.1 macro F1 in the emotion of generated speech as judged by an automated audio model. If true, the result means vision is not a redundant channel for expressive speech: facial expression supplies information the voice alone does not.

Core claim

The paper claims that conditioning a pre-trained expressive speech language model on full-face visual features, and not just lip-region video, improves both the model's ability to identify a speaker's emotion and its ability to produce spoken responses whose emotion matches the context. The recipe is two-stage: first pre-train a Q-Former-style fusion module that compresses visual frames into query latents prepended to speech tokens, using next-token prediction on unlabeled video; then fine-tune on emotion classification and expressive dialogue built from IEMOCAP, with an auxiliary emotion classifier that supplies the generation prompt's emotion label at inference. In the head-to-head compari

What carries the argument

The central mechanism is the Q-Former Prefix fusion: query latents attend to visual features through cross-attention and are prepended to the SpeechLM's speech-token sequence, so every generation step can condition on compressed visual information. The visual stream is encoded by SMIRK, a lightweight 3D face-reconstruction model whose expressive and jaw parameters are used instead of raw pixels or lip crops. The speech stream is tokenized into interleaved semantic, style, and pitch tokens by the base expressive SpeechLM's tokenizer, and LoRA updates adapt the model during pre-training and fine-tuning. An auxiliary emotion classifier over visual-query and style/pitch hidden states provides th

Load-bearing premise

The reported gains rest on the premise that the synthetic IEMOCAP-derived dialogues and the automated emotion judgments on generated audio behave like real expressive human conversation; if the model is only mimicking the TTS voice it was trained on, the visual advantage may not appear with real speakers.

What would settle it

Fine-tune the same two models on real, non-synthetic expressive video dialogue and have human listeners label the emotion of the generated responses; if the audio-visual model does not beat the speech-only model on human-rated emotion F1, the central claim is not supported. A cheaper check: replace the video with static frames at evaluation; if the emotion-recognition F1 gain persists, the model is not using facial dynamics.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Emotion-aware conversational agents can condition spoken responses on the speaker's face, not just the voice, improving emotional alignment without retraining a speech model from scratch.
  • Video-conditioned pre-training also helps audio-visual speech recognition, lowering word error rate under noise and speech-token masking compared with the speech-only model.
  • The Q-Former Prefix with 3D facial-expression features is a transferable recipe for adding vision to an existing speech language model while keeping next-token prediction as the objective.
  • The emotion label alone is a weak control: with in-context demonstrations the model shifts generated emotion, but without them it follows the audio-visual input.
  • Expressive dialogue data can be synthetically expanded from small acted corpora by rewriting short turns and synthesizing longer responses with voice cloning.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A human listening test on held-out real dialogue would be the sharper check on the expressiveness gain; a single automated audio judge could share systematic biases with the TTS used for training data.
  • Because the visual features are full-face rather than lip-cropped, the gain could come from expression reading rather than articulation; replacing video with static frames during evaluation would separate those two sources.
  • The weak zero-shot controllability suggests the model anchors emotion to the input speaker's face and voice; training data that decouples input and response emotion could make label-based style control work without in-context examples.
  • If the visual benefit transfers to noisy or bandwidth-limited audio, the same architecture could improve emotion-aware speech systems where the audio channel is degraded but a webcam feed is available.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes an Audio-Visual Language Model (AVLM) that integrates full-face visual features into a pre-trained expressive SpeechLM (SpiritLM) via a Q-Former prefix module. Training proceeds in two stages: (i) self-supervised pretraining on LRS3 with next-token prediction over speech, style, and pitch tokens, comparing visual encoders (SMIRK, VGG-Face2, MAGVIT2) and fusion strategies; and (ii) fine-tuning on IEMOCAP-derived synthetic expressive dialogues for emotion classification and expressive speech response generation. The reported results are: lower pretraining perplexity for the chosen prefix model (5.5 vs 5.6), improved AVSR WER under clean/noisy conditions (Table 5), higher emotion-recognition F1 (66.2 vs 61.3 in Table 7), and higher Qwen2-Audio-judged emotional F1 for generated speech (42.49 vs 38.39 in Table 7). The central claim is that visual cues improve both emotion recognition and emotional expressiveness of generated speech.

Significance. If the central claim were established, this would be a useful contribution: most expressive speech LMs are audio-only, and the paper systematically explores full-face visual features rather than lip-only features, with code released and several external benchmarks used for evaluation. The pretraining and AVSR results provide converging evidence that the visual prefix is learned and that it helps under acoustic degradation. The emotion-recognition gain on IEMOCAP is also consistent with prior audio-visual affect literature. However, the headline generation result — the part that would make this a foundation for emotionally intelligent conversational agents — currently rests on a confounded comparison and a synthetic training/evaluation loop. The paper itself notes the lack of human evaluation and the small amount of data. The contribution is therefore defensible in direction but not yet fully supported in its strongest form.

major comments (3)
  1. [§5.3 / Appendix D, Table 7] The generation comparison in Table 7 is confounded by the emotion-label conditioning. At inference, Appendix D states that the emotion label in the generation prompt is obtained from each model's own auxiliary classifier. The AVLM classifier is 4.9 F1 points better than the speech-only classifier (66.2 vs 61.3), so the AVLM generation prompt is conditioned on a more accurate label. The +4.1 F1 generation gain (42.49 vs 38.39) can therefore be explained by label accuracy rather than by the visual prefix. This is not a minor issue because Fig. 5 shows that the label influences the output. Please add a controlled experiment in which both models are conditioned on the same gold labels, or on the same fixed predicted labels, and report generation F1 separately. A human listening study is also needed; the Limitations section already concedes that no human evaluation is included.
  2. [Appendix A / §5.3] The synthetic training and evaluation loop makes the generation result hard to interpret. The target response audio is synthesized by Step-Audio-TTS-3B conditioned on the same IEMOCAP emotion label and a cloned speaker voice; these same synthetic responses are used as training targets and are scored by Qwen2-Audio. A model that predicts the emotion label accurately can imitate Step-Audio's label-to-prosody mapping, and Qwen2-Audio may recognize that synthetic style rather than natural expressiveness. Thus the reported gains do not yet show that full-face visual cues improve expressive speech generation beyond improving emotion-label prediction in this synthetic setting. Please evaluate on original IEMOCAP response audio or another natural expressive audio-visual dialogue set, and complement with human perceptual judgments.
  3. [Table 7 / Appendix D] The main evaluation has no error bars, confidence intervals, or significance tests, and the test set is 10% of 4,859 synthetic pairs (about 486 utterances). A 4–5 F1 gap on this size can be within sampling noise, especially for four-way macro-F1 of generated speech judged by a third-party audio model. Relatedly, the Qwen2-Audio prompt in Appendix D allows 'Frustrated' as an output label, while the task and ground truth have only four classes (Angry and Frustrated are merged in Appendix A). The paper does not specify how 'Frustrated' predictions are mapped or excluded. Please report multiple seeds or bootstrap confidence intervals and define the label post-processing.
minor comments (6)
  1. [Abstract / Table 7] The abstract says '+5 F1 in emotion recognition', but Table 7 reports 66.2 vs 61.3, which is +4.9. Use consistent rounding.
  2. [§5.3] There is a typo: 'style/pitch tokens posii' should likely be 'positions'.
  3. [Appendix D] The decoding procedure is described in prose. A short pseudocode or explicit transition table would make the constrained decoding reproducible.
  4. [Appendix C] The text says clips with average yaw angle exceeding '30%' are filtered; this should be '30 degrees'.
  5. [§5.1, Table 3] DIRECT CONCAT yields PPL 132.0, which is dramatically worse than the baselines; the paper attributes this to adaptation difficulty. A brief explanation of whether this is convergence failure or a structural issue would be helpful.
  6. [Appendix A] For Step-Audio synthesis, the exact style-control prompt is '[Emotion] [Text Response]'. It would be useful to state whether any additional style parameters (e.g., speed, pitch) were used, since the synthetic corpus is the basis for all fine-tuning.

Circularity Check

0 steps flagged

No significant circularity: core comparisons rest on external IEMOCAP labels and Qwen2-Audio, and no fitted parameter or self-citation is presented as a prediction.

full rationale

The paper's derivation chain is self-contained against external benchmarks. Pre-training is evaluated by perplexity on LRS3 and by AVSR WER on LRS3 transcriptions; both are external ground truths. Emotion recognition is evaluated against IEMOCAP labels, and the generation evaluation uses Qwen2-Audio, a third-party model, again against external IEMOCAP-derived labels. No step reduces by construction to its own inputs. The synthetic data loop (Appendix A) uses the emotion label to drive Step-Audio TTS and then the same label as ground truth, but this is a data-generation choice, not a logical identity: the model must still generate new speech tokens, and the AVLM-vs-speech-only comparison is not statistically forced. The inference-time conditioning on each model's own predicted emotion label is a potential confound for the generation F1 comparison, since AVLM has a better classifier; however, the paper's own controllability analysis (Fig. 5, no-ICL panel) indicates the inserted label has limited control in the zero-shot setting used for Table 7, so the generation advantage cannot be reduced to label accuracy by construction. Self-citations are limited to related-work mentions and are not load-bearing; no uniqueness theorem or ansatz is smuggled from prior work by the same authors. The limitations section appropriately concedes the lack of human evaluation and the small amount of fine-tuning data, which are external-validity concerns rather than circularity. Therefore no circular step is identifiable.

Axiom & Free-Parameter Ledger

5 free parameters · 6 axioms · 0 invented entities

The central claim rests on domain assumptions about the reliability of synthetic emotion data and proxy emotion evaluation, plus a set of hand-chosen hyperparameters. No unproved mathematical axioms beyond standard attention operations are needed, and no new physical entity or dimension is postulated.

free parameters (5)
  • Q-Former prefix compression ratio = 5 visual query latents per second at 25 fps
    Controls the amount of visual information retained; set by hand in Appendix D and not swept.
  • pretraining attention masking ratio = 30%
    Selected as best among {0, 30, 50, 70} by PPL in Table 4; used for all downstream fine-tuning.
  • SMIRK feature subset = 55-dim expression + 3-dim jaw = 58 dims
    Chosen by hand as the most emotion- and lip-relevant FLAME parameters in Appendix C; other shape, pose, and camera parameters discarded.
  • LoRA rank/alpha/dropout = r=16, alpha=32, dropout=0.05
    Standard PEFT hyperparameters; no sweep reported, used in all fine-tuning experiments.
  • head-yaw filter threshold = 30 degrees average yaw
    Ad hoc data cleaning rule in Appendix C that removes about 10% of training clips; no ablation of the threshold is reported.
axioms (6)
  • domain assumption Visual modality provides complementary non-semantic emotional information beyond speech.
    Premise of Section 3 and the whole approach; supported by the motivation experiments but not proven across domains.
  • domain assumption The synthetic response audio from Step-Audio, conditioned on an emotion label and cloned voice, is a faithful stand-in for real expressive dialogue audio.
    Needed for the fine-tuning and evaluation in Section 5.3; if the synthetic style dominates, the F1 gains may not transfer to real speech.
  • domain assumption Input and response speech in each IEMOCAP-derived conversation share the same emotion label.
    Explicitly stated in Section 4.2; used to assign ground-truth emotion for both recognition and generation targets.
  • domain assumption Qwen2-Audio's emotion labels are a valid proxy for human perception of generated speech expressiveness.
    Used in Section 5.3 and Appendix D for the speech-generation metric; no human listening test is provided.
  • standard math Cross-attention with softmax is a valid differentiable alignment operation.
    Standard transformer machinery invoked in Equation 1, not re-derived in the paper.
  • ad hoc to paper Angry and Frustrated emotion labels can be merged into one class.
    Data processing decision in Appendix A that changes label granularity; not justified with ablations.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Seeing is Believing: Emotion-Aware Audio-Visual Language Modeling for Expressive Speech Generation." pith.science (2026). https://pith.science/paper/VH5RUQQW

@misc{pith2026250816188,
  author       = {Pith},
  title        = {Pith review of: Seeing is Believing: Emotion-Aware Audio-Visual Language Modeling for Expressive Speech Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VH5RUQQW}},
  note         = {Machine review of arXiv:2508.16188}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

We present an Audio-Visual Language Model (AVLM) for expressive speech generation by integrating full-face visual cues into a pre-trained expressive speech model. We explore multiple visual encoders and multimodal fusion strategies during pre-training to identify the most effective integration approach. Subsequent fine-tuning on emotion recognition and expressive dialogue tasks yields substantial gains over speech-only baselines (e.g., +5 F1 in emotion recognition). AVLM highlights the value of expressive visual information in guiding speech generation and offers a foundation for end-to-end multimodal conversational systems.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

55 extracted references · 21 canonical work pages · 6 internal anchors

  1. [1]

    Triantafyllos Afouras, Joon Son Chung, and Andrew Zisserman. 2018. https://arxiv.org/abs/1809.00496 Lrs3-ted: a large-scale dataset for visual speech recognition . Preprint, arXiv:1809.00496

    Pith/arXiv arXiv 2018

  2. [2]

    Keyu An, Qian Chen, Chong Deng, Zhihao Du, Changfeng Gao, Zhifu Gao, Yue Gu, Ting He, Hangrui Hu, Kai Hu, Shengpeng Ji, Yabin Li, Zerui Li, Heng Lu, Haoneng Luo, Xiang Lv, Bin Ma, Ziyang Ma, Chongjia Ni, and 14 others. 2024. https://arxiv.org/abs/2407.04051 Funaudiollm: Voice understanding and generation foundation models for natural interaction between h...

    Pith/arXiv arXiv 2024

  3. [3]

    Mohamed Anwar, Bowen Shi, Vedanuj Goswami, Wei-Ning Hsu, Juan Pino, and Changhan Wang. 2023. https://arxiv.org/abs/2303.00628 Muavic: A multilingual audio-visual corpus for robust speech recognition and robust speech-to-text translation . Preprint, arXiv:2303.00628

    Pith/arXiv arXiv 2023

  4. [4]

    Chang, Sungbok Lee, and Shrikanth S

    Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Ebrahim (Abe) Kazemzadeh, Emily Mower Provost, Samuel Kim, Jeannette N. Chang, Sungbok Lee, and Shrikanth S. Narayanan. 2008. https://api.semanticscholar.org/CorpusID:11820063 Iemocap: interactive emotional dyadic motion capture database . Language Resources and Evaluation, 42:335--359

    work page 2008

  5. [5]

    Carlos Busso, Srinivas Parthasarathy, Alec Burmania, Mohammed AbdelWahab, Najmeh Sadoughi, and Emily Mower Provost. 2017. https://doi.org/10.1109/TAFFC.2016.2515617 Msp-improv: An acted corpus of dyadic interactions to study emotion perception . IEEE Transactions on Affective Computing, 8(1):67--80

    work page arXiv 2017

  6. [6]

    Cooper, Michael K

    Houwei Cao, David G. Cooper, Michael K. Keutmann, Ruben C. Gur, Ani Nenkova, and Ragini Verma. 2014. https://doi.org/10.1109/TAFFC.2014.2336244 Crema-d: Crowd-sourced emotional multimodal actors dataset . IEEE Transactions on Affective Computing, 5(4):377--390

    work page arXiv 2014

  7. [7]

    Parkhi, and Andrew Zisserman

    Qiong Cao, Li Shen, Weidi Xie, Omkar M. Parkhi, and Andrew Zisserman. 2018. https://arxiv.org/abs/1710.08092 Vggface2: A dataset for recognising faces across pose and age . Preprint, arXiv:1710.08092

    Pith/arXiv arXiv 2018

  8. [8]

    Umberto Cappellazzo, Minsu Kim, Honglie Chen, Pingchuan Ma, Stavros Petridis, Daniele Falavigna, Alessio Brutti, and Maja Pantic. 2025. https://arxiv.org/abs/2409.12319 Large language models are strong audio-visual speech recognition learners . Preprint, arXiv:2409.12319

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  9. [9]

    Qian Chen, Yafeng Chen, Yanni Chen, Mengzhe Chen, Yingda Chen, Chong Deng, Zhihao Du, Ruize Gao, Changfeng Gao, Zhifu Gao, Yabin Li, Xiang Lv, Jiaqing Liu, Haoneng Luo, Bin Ma, Chongjia Ni, Xian Shi, Jialong Tang, Hui Wang, and 17 others. 2025. https://arxiv.org/abs/2501.06282 Minmo: A multimodal large language model for seamless voice interaction . Prepr...

    Pith/arXiv arXiv 2025

  10. [10]

    Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, Chang Zhou, and Jingren Zhou. 2024. https://arxiv.org/abs/2407.10759 Qwen2-audio technical report . Preprint, arXiv:2407.10759

    Pith/arXiv arXiv 2024

  11. [11]

    Joon Son Chung, Arsha Nagrani, and Andrew Zisserman. 2018. https://doi.org/10.21437/Interspeech.2018-1929 Voxceleb2: Deep speaker recognition . In Interspeech 2018, pages 1086--1090

    doi:10.21437/interspeech.2018-1929 2018

  12. [12]

    Zhihao Du, Yuxuan Wang, Qian Chen, Xian Shi, Xiang Lv, Tianyu Zhao, Zhifu Gao, Yexin Yang, Changfeng Gao, Hui Wang, Fan Yu, Huadai Liu, Zhengyan Sheng, Yue Gu, Chong Deng, Wen Wang, Shiliang Zhang, Zhijie Yan, and Jingren Zhou. 2024. https://arxiv.org/abs/2412.10117 Cosyvoice 2: Scalable streaming speech synthesis with large language models . Preprint, ar...

    Pith/arXiv arXiv 2024

  13. [13]

    Paul-Ambroise Duquenne, Kevin Heffernan, Alexandre Mourachko, Beno \^i t Sagot, and Holger Schwenk. 2023. https://hal.science/hal-04629427 SONAR EXPRESSIVE: Zero-shot Expressive Speech-to-Speech Translation . Working paper or preprint

    work page 2023

  14. [14]

    Alexandre Défossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi. 2022. https://arxiv.org/abs/2210.13438 High fidelity neural audio compression . Preprint, arXiv:2210.13438

    Pith/arXiv arXiv 2022

  15. [15]

    Zhifu Gao, Shiliang Zhang, Ian McLoughlin, and Zhijie Yan. 2022. https://doi.org/10.21437/INTERSPEECH.2022-9996 Paraformer: Fast and accurate parallel transformer for non-autoregressive end-to-end speech recognition . In 23rd Annual Conference of the International Speech Communication Association, Interspeech 2022, Incheon, Korea, September 18-22, 2022, p...

    doi:10.21437/interspeech.2022-9996 2022

  16. [16]

    Alexandros Haliassos, Rodrigo Mira, Honglie Chen, Zoe Landgraf, Stavros Petridis, and Maja Pantic. 2024. https://proceedings.neurips.cc/paper_files/paper/2024/file/fc69c48e41f7143e2e570f68c0b0951c-Paper-Conference.pdf Unified speech recognition: A single model for auditory, visual, and audiovisual inputs . In Advances in Neural Information Processing Syst...

    work page 2024

  17. [17]

    HyoJung Han, Mohamed Anwar, Juan Pino, Wei-Ning Hsu, Marine Carpuat, Bowen Shi, and Changhan Wang. 2024. https://doi.org/10.18653/v1/2024.acl-long.697 XLAVS - R : Cross-lingual audio-visual speech representation learning for noise-robust speech perception . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume ...

    doi:10.18653/v1/2024.acl-long.697 2024

  18. [18]

    Joanna Hong, Minsu Kim, Jeong Yun Choi, and Yong Man Ro. 2023. https://api.semanticscholar.org/CorpusID:257532934 Watch or listen: Robust audio-visual speech recognition with visual corruption modeling and reliability scoring . 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 18783--18794

    work page 2023

  19. [19]

    Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. 2021. https://arxiv.org/abs/2106.07447 Hubert: Self-supervised speech representation learning by masked prediction of hidden units . Preprint, arXiv:2106.07447

    Pith/arXiv arXiv 2021

  20. [20]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. https://arxiv.org/abs/2106.09685 Lora: Low-rank adaptation of large language models . Preprint, arXiv:2106.09685

    Pith/arXiv arXiv 2021

  21. [21]

    Ailin Huang, Boyong Wu, Bruce Wang, Chao Yan, Chen Hu, Chengli Feng, Fei Tian, Feiyu Shen, Jingbei Li, Mingrui Chen, Peng Liu, Ruihang Miao, Wang You, Xi Chen, Xuerui Yang, Yechang Huang, Yuxiang Zhang, Zheng Gong, Zixin Zhang, and 126 others. 2025. https://arxiv.org/abs/2502.11946 Step-audio: Unified understanding and generation in intelligent speech int...

    Pith/arXiv arXiv 2025

  22. [22]

    LMCodec: A Low Bitrate Speech Codec With Causal Transformer Models

    Teerapat Jenrungrot, Michael Chinen, W. Bastiaan Kleijn, Jan Skoglund, Zalán Borsos, Neil Zeghidour, and Marco Tagliasacchi. 2023. https://arxiv.org/abs/2303.12984 Lmcodec: A low bitrate speech codec with causal transformer models . Preprint, arXiv:2303.12984

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  23. [23]

    Eugene Kharitonov, Ann Lee, Adam Polyak, Yossi Adi, Jade Copet, Kushal Lakhotia, Tu Anh Nguyen, Morgane Riviere, Abdelrahman Mohamed, Emmanuel Dupoux, and Wei-Ning Hsu. 2022. https://doi.org/10.18653/v1/2022.acl-long.593 Text-free prosody-aware generative spoken language modeling . In Proceedings of the 60th Annual Meeting of the Association for Computati...

    work page doi:10.18653/v1/2022.acl-long.593 2022

  24. [24]

    Charles, and 21 others

    KimiTeam, Ding Ding, Zeqian Ju, Yichong Leng, Songxiang Liu, Tong Liu, Zeyu Shang, Kai Shen, Wei Song, Xu Tan, Heyi Tang, Zhengtao Wang, Chu Wei, Yifei Xin, Xinran Xu, Jianwei Yu, Yutao Zhang, Xinyu Zhou, Y. Charles, and 21 others. 2025. https://arxiv.org/abs/2504.18425 Kimi-audio technical report . Preprint, arXiv:2504.18425

    Pith/arXiv arXiv 2025

  25. [25]

    Nicolaou, Athanasios Papaioannou, Guoying Zhao, Björn Schuller, Irene Kotsia, and Stefanos Zafeiriou

    Dimitrios Kollias, Panagiotis Tzirakis, Mihalis A. Nicolaou, Athanasios Papaioannou, Guoying Zhao, Björn Schuller, Irene Kotsia, and Stefanos Zafeiriou. 2019. https://doi.org/10.1007/s11263-019-01158-4 Deep affect prediction in-the-wild: Aff-wild database and challenge, deep architectures, and beyond . International Journal of Computer Vision, 127(6–7):907–929

    work page doi:10.1007/s11263-019-01158-4 2019

  26. [26]

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. https://arxiv.org/abs/2301.12597 Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models . Preprint, arXiv:2301.12597

    Pith/arXiv arXiv 2023

  27. [27]

    Tianye Li, Timo Bolkart, Michael. J. Black, Hao Li, and Javier Romero. 2017. https://doi.org/10.1145/3130800.3130813 Learning a model of facial shape and expression from 4D scans . ACM Transactions on Graphics, (Proc. SIGGRAPH Asia), 36(6):194:1--194:17

    arXiv 2017

  28. [28]

    Steven R Livingstone and Frank A Russo. 2018. The ryerson audio-visual database of emotional speech and song (ravdess): A dynamic, multimodal set of facial and vocal expressions in north american english. PloS one, 13(5):e0196391

    work page 2018

  29. [29]

    Zhuoyan Luo, Fengyuan Shi, Yixiao Ge, Yujiu Yang, Limin Wang, and Ying Shan. 2025. https://arxiv.org/abs/2409.04410 Open-magvit2: An open-source project toward democratizing auto-regressive visual generation . Preprint, arXiv:2409.04410

    Pith/arXiv arXiv 2025

  30. [30]

    Hui Ma, Jian Wang, Hongfei Lin, Bo Zhang, Yijia Zhang, and Bo Xu. 2024 a . https://doi.org/10.1109/TMM.2023.3271019 A transformer-based model with self-distillation for multimodal emotion recognition in conversations . IEEE Transactions on Multimedia, 26:776--788

    work page arXiv 2024

  31. [31]

    Ziyang Ma, Mingjie Chen, Hezhao Zhang, Zhisheng Zheng, Wenxi Chen, Xiquan Li, Jiaxin Ye, Xie Chen, and Thomas Hain. 2024 b . https://api.semanticscholar.org/CorpusID:270380401 Emobox: Multilingual multi-corpus speech emotion recognition toolkit and benchmark . ArXiv, abs/2406.07162

    Pith/arXiv arXiv 2024

  32. [32]

    Albert Mehrabian and James A Russell. 1974. An Approach to Environmental Psychology. MIT Press

    work page 1974

  33. [33]

    Costa-jussa, Maha Elbayad, Sravya Popuri, Paul-Ambroise Duquenne, Robin Algayres, Ruslan Mavlyutov, Itai Gat, Gabriel Synnaeve, Juan Pino, Benoit Sagot, and Emmanuel Dupoux

    Tu Anh Nguyen, Benjamin Muller, Bokai Yu, Marta R. Costa-jussa, Maha Elbayad, Sravya Popuri, Paul-Ambroise Duquenne, Robin Algayres, Ruslan Mavlyutov, Itai Gat, Gabriel Synnaeve, Juan Pino, Benoit Sagot, and Emmanuel Dupoux. 2024. https://arxiv.org/abs/2402.05755 Spirit-lm: Interleaved spoken and written language model . Preprint, arXiv:2402.05755

    Pith/arXiv arXiv 2024

  34. [34]

    OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, and 262 others. 2024. https://arxiv.org/abs/2303.08774 Gpt-4 technical r...

    Pith/arXiv arXiv 2024

  35. [35]

    Adam Polyak, Yossi Adi, Jade Copet, Eugene Kharitonov, Kushal Lakhotia, Wei-Ning Hsu, Abdelrahman Mohamed, and Emmanuel Dupoux. 2021. https://arxiv.org/abs/2104.00355 Speech resynthesis from discrete disentangled self-supervised representations . Preprint, arXiv:2104.00355

    Pith/arXiv arXiv 2021

  36. [36]

    Gnana Praveen, Patrick Cardinal, and Eric Granger

    R. Gnana Praveen, Patrick Cardinal, and Eric Granger. 2023. https://doi.org/10.1109/TBIOM.2022.3233083 Audio–visual fusion for emotion recognition in the valence–arousal space using joint cross-attention . IEEE Transactions on Biometrics, Behavior, and Identity Science, 5(3):360--373

    work page arXiv 2023

  37. [37]

    Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2022. https://arxiv.org/abs/2212.04356 Robust speech recognition via large-scale weak supervision . Preprint, arXiv:2212.04356

    Pith/arXiv arXiv 2022

  38. [38]

    Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. 2016. https://arxiv.org/abs/1506.02640 You only look once: Unified, real-time object detection . Preprint, arXiv:1506.02640

    Pith/arXiv arXiv 2016

  39. [39]

    Filntisis, Radek Danecek, Victoria F

    George Retsinas, Panagiotis P. Filntisis, Radek Danecek, Victoria F. Abrevaya, Anastasios Roussos, Timo Bolkart, and Petros Maragos. 2024. 3d facial expressions through analysis-by-neural-synthesis. In Conference on Computer Vision and Pattern Recognition (CVPR)

    work page 2024

  40. [40]

    Fabien Ringeval, Andreas Sonderegger, J \"u rgen Sauer, and Denis Lalanne. 2013. https://diuf.unifr.ch/main/diva/recola/ Introducing the RECOLA multimodal corpus of remote collaborative and affective interactions . In Proceedings of the 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), pages 1--8. IEEE

    work page 2013

  41. [41]

    Savchenko

    Andrey V. Savchenko. 2022. https://doi.org/10.1016/j.simpa.2022.100433 Hsemotion: High-speed emotion recognition library . Software Impacts, 14:100433

    work page arXiv 2022

  42. [42]

    Bowen Shi, Wei-Ning Hsu, Kushal Lakhotia, and Abdelrahman Mohamed. 2022. https://arxiv.org/abs/2201.02184 Learning audio-visual speech representation by masked multimodal cluster prediction . Preprint, arXiv:2201.02184

    Pith/arXiv arXiv 2022

  43. [43]

    Weiting Tan, Hirofumi Inaguma, Ning Dong, Paden Tomasello, and Xutai Ma. 2024. https://arxiv.org/abs/2410.00168 Ssr: Alignment-aware modality connector for speech language models . Preprint, arXiv:2410.00168

    Pith/arXiv arXiv 2024

  44. [44]

    Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, and Chao Zhang. 2024 a . https://arxiv.org/abs/2310.13289 Salmonn: Towards generic hearing abilities for large language models . Preprint, arXiv:2310.13289

    Pith/arXiv arXiv 2024

  45. [45]

    Haobin Tang, Xulong Zhang, Ning Cheng, Jing Xiao, and Jianzong Wang. 2024 b . https://arxiv.org/abs/2401.08166 Ed-tts: Multi-scale emotion modeling using cross-domain emotion diarization for emotional speech synthesis . Preprint, arXiv:2401.08166

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  46. [46]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf Attention is all you need . In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc

    2017

  47. [47]

    Shijun Wang, Jón Guðnason, and Damian Borth. 2023. https://arxiv.org/abs/2306.05709 Learning emotional representations from imbalanced speech data for speech emotion recognition and emotional text-to-speech . Preprint, arXiv:2306.05709

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  48. [48]

    Jian Wu, Yashesh Gaur, Zhuo Chen, Long Zhou, Yimeng Zhu, Tianrui Wang, Jinyu Li, Shujie Liu, Bo Ren, Linquan Liu, and Yu Wu. 2023. https://arxiv.org/abs/2307.03917 On decoder-only architecture for speech-to-text and large language model integration . Preprint, arXiv:2307.03917

    Pith/arXiv arXiv 2023

  49. [49]

    Jeong Hun Yeo, Minsu Kim, Chae Won Kim, Stavros Petridis, and Yong Man Ro. 2025 a . https://arxiv.org/abs/2503.06273 Zero-avsr: Zero-shot audio-visual speech recognition with llms by learning language-agnostic speech representations . Preprint, arXiv:2503.06273

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  50. [50]

    Jeong Hun Yeo, Hyeongseop Rha, Se Jin Park, and Yong Man Ro. 2025 b . https://arxiv.org/abs/2503.11315 Mms-llama: Efficient llm-based audio-visual speech recognition with minimal multimodal speech tokens . Preprint, arXiv:2503.11315

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  51. [51]

    Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Vighnesh Birodkar, Agrim Gupta, Xiuye Gu, Alexander G

    Lijun Yu, José Lezama, Nitesh B. Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Vighnesh Birodkar, Agrim Gupta, Xiuye Gu, Alexander G. Hauptmann, Boqing Gong, Ming-Hsuan Yang, Irfan Essa, David A. Ross, and Lu Jiang. 2024. https://arxiv.org/abs/2310.05737 Language model beats diffusion -- tokenizer is key to visual generation . Preprint...

    Pith/arXiv arXiv 2024

  52. [52]

    Wenyi Yu, Changli Tang, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, and Chao Zhang. 2023. https://arxiv.org/abs/2309.13963 Connecting speech encoder and large language model for asr . Preprint, arXiv:2309.13963

    Pith/arXiv arXiv 2023

  53. [53]

    Dong Zhang, Shimin Li, Xin Zhang, Jun Zhan, Pengyu Wang, Yaqian Zhou, and Xipeng Qiu. 2023. https://arxiv.org/abs/2305.11000 Speechgpt: Empowering large language models with intrinsic cross-modal conversational abilities . Preprint, arXiv:2305.11000

    Pith/arXiv arXiv 2023

  54. [54]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  55. [55]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.