Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

EmoSLLM: Parameter-Efficient Adaptation of LLMs for Speech Emotion Recognition

T0 review · 4 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that a 3.2B-parameter speech-text LLM, fine-tuned with only 12.5M trainable parameters, outperforms all but one existing 7B-parameter rival on MSP-Podcast speech emotion recognition.

desk verdict The headline accuracy is earned by giving the model the ground-truth transcript at inference; without that privilege the model drops to 0.417–0.431, so the 'outperforms all but one' claim is not apples-to-apples. read the letter →

arxiv 2508.14130 v1 pith:VQYXBENW submitted 2025-08-19 eess.AS cs.LG

Hugo Thimonier , Antony Perzo , Renaud Seguier This is my paper
classification eess.AScs.LG
keywords speechemotionrecognitionlargelanguagemodelsLoRAparameter-efficientfine-tuningmultimodalaudio-textquerypoolingmapperMSP-PodcastWavLM
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

EmoSLLM is a speech-text large language model built for speech emotion recognition, and this paper argues that it is strong enough to beat most far larger rivals despite a tiny trainable footprint. The model freezes a WavLM audio encoder and a Llama 3.2-3B backbone, maps audio features into the LLM space with a QPMapper downsampling module, and adapts the LLM with LoRA through a three-phase curriculum that starts with speech recognition and ends with joint transcription-plus-emotion prediction. On the MSP-Podcast test1 split it achieves 49.7% unweighted accuracy, outperforming SALMONN, Qwen2-Audio, and OASQAL while trailing only SIFT-LLM, and does so with roughly a third of the compute of the cheapest baseline. The central claim is that parameter-efficient adaptation is enough to make LLMs practical for on-device, privacy-sensitive emotion monitoring.

What carries the argument

QPMapper (Query Pooling Mapper): a small transformer with 32 learnable queries that downsamples the audio encoder's sequence to a fixed length and projects it into the LLM's embedding space. LoRA adapters on all attention and MLP projections of the LLM allow parameter-efficient fine-tuning. The three-phase curriculum—frozen-LLM ASR alignment, then LoRA-adapted ASR, then joint ASR+SER—plus the joint-decoding prompt (| ASR: <transcript> | Emotion:) is what carries the emotion-recognition performance.

What would settle it

Run inference with an off-the-shelf ASR system (e.g., Whisper) producing the transcript instead of the gold transcript, then measure SER accuracy on MSP-Podcast test1. If accuracy drops toward the 41.7% SER-only baseline, the reported 49.7% depends on oracle text and the comparison with transcript-free baselines is not apples-to-apples.

Watch

Extended reading notes

Core claim

The paper's claim is that a 3.2B-parameter LLM with 12.5M trainable parameters can match or beat 7B+ speech-text LLMs at speech emotion recognition when the audio is fed in through a learnable query-pooling mapper and the LLM is adapted with LoRA. The reported 49.7% accuracy on MSP-Podcast test1 is produced by joint decoding: the model sees the audio tokens and the true transcript, then generates the emotion label. The paper shows that removing the transcript cue drops accuracy to 41.7%, while adding binned paralinguistic features and a 1-shot format hint each add about one to 1.5 points. The authors attribute the remaining gap to SIFT-LLM's larger backbone (Qwen2.5-7B) and much larger multi

Load-bearing premise

The headline result assumes the true speech transcript is available and inserted into the prompt at inference time; without that transcript, accuracy drops from 49.7% to 41.7%.

Editorial extensions

If this is right

  • A speech-text LLM of 3.2B parameters is enough for competitive SER, suggesting on-device emotion monitoring is within reach.
  • The three-phase curriculum (ASR alignment before emotion training) is a reusable recipe for other audio-conditioned LLM tasks.
  • Providing the transcript inside the assistant's answer rather than in the user prompt matters: same info, 6.6 points difference (0.497 vs 0.431).
  • Adding binned paralinguistic features and 1-shot format hints yields cheap accuracy gains (1.1 and 1.5 points respectively) without retraining the audio encoder.
  • Competitive performance at lower parameter count implies comparisons in the field should report parameter counts alongside accuracy.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The accuracy gain from the transcript hint is measured against baselines that do not receive the transcript; a fair comparison would give transcript hints (or ASR predictions) to all models or none.
  • Since the model was never trained on user prompts containing transcripts, robustness to ASR errors is untested; inserting an off-the-shelf ASR transcript at inference is an empirical test of whether the gain survives real conditions.
  • The paralinguistic features (loudness, pitch, jitter, shimmer) are supplied as text; an LLM that could read them from the audio tokens directly would remove the need for an external feature extractor at deployment.
  • The recipe should transfer to other audio-understanding tasks (e.g., speaker verification, audio captioning) where a small trainable adapter on a 3B backbone could replace 7B+ multi-task models.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes EmoSLLM, a parameter-efficient speech emotion recognition (SER) system that adapts Llama-3.2-3B-Instruct through a QPMapper downsampling module and LoRA adapters. Training follows a three-stage curriculum: (P1) ASR alignment with frozen audio encoder and LLM, (P2) ASR with LoRA, and (P3) joint ASR+SER on MSP-Podcast and Librispeech. Evaluation on MSP-Podcast test1 reports 0.497 accuracy with 12.5M trainable parameters and 3.2B total parameters. The authors claim this outperforms all but one existing Speech-Text LLM (SIFT-LLM) while using less than half the parameters. Ablations examine the effect of paralinguistic features, few-shot hinting, joint decoding with transcript hints at inference, and choice of audio encoder.

Significance. If the central comparison were fair, the result would be a useful step toward on-device SER: the QPMapper+LoRA recipe is technically sound, the three-stage curriculum is clearly presented, and the 12.5M trainable parameter count is attractive. The paper also reports substantial compute savings relative to the baselines. However, the headline result depends critically on a nonstandard inference protocol in which the model receives the ground-truth transcript, while the comparison baselines are evaluated without it. The paper contains no error bars or statistical tests, and one announced ablation (Section 5.1) is missing entirely. These issues make the significance contingent on experiments that are not currently reported.

major comments (4)
  1. [§5.3, Table 3; Abstract; §4.2] The 0.497 accuracy used in the abstract and §4.2 is obtained by the 'EmoSLLM' inference protocol, where the ground-truth transcript is prepended to the assistant answer as '| ASR: <transcript> | Emotion:' before generation. The comparison baselines (SALMONN, Qwen2-Audio, OASQA-LLM, SIFT-LLM) are not given this information. The paper's own ablations quantify the effect: SER-only scores 0.417 and Prompt-hint scores 0.431. Unless the baselines are also evaluated with the same transcript hint, the claim 'outperforms all but one existing Speech-Text LLMs' is not supported by the reported comparison. This is the load-bearing assumption of the paper, and it fails as stated.
  2. [§5.1] Section 5.1 states that the authors compare Qwen3-4B and Llama 3.2-3B-Instruct as the backbone LLM, but no results are given for this ablation. The section contains only two sentences and no table or figure. This omitted result prevents verification of the effect of the backbone LLM and should either be added or the section removed.
  3. [Tables 1–4] None of the reported accuracies are accompanied by error bars, multiple seeds, confidence intervals, or significance tests. For instance, Table 2 shows 0.473 (1-shot) vs 0.474 (2-shot), and Table 1 shows 0.458 vs 0.469; these differences may be within run-to-run noise. The conclusions in §5.2 and §5.3, including the gain from transcript-based prompting (0.417–0.497), rely on single-run comparisons. Repeated runs or bootstrap intervals are needed before these differences can be interpreted.
  4. [§4.2, §5.3] The parameter-efficiency claim ('less than half the parameters of competing approaches') is incomplete. The EmoSLLM inference protocol requires a transcript for every test sample; in practice this transcript must be generated by an external ASR model, whose parameters and compute are not counted in the reported 12.5M trainable / 3.2B total footprint. The comparison should either include an ASR model in the total parameter count or use a fair protocol in which all models receive the same transcript information. As written, the efficiency story overstates the deployment-ready footprint.
minor comments (5)
  1. [§3.2 vs §3.4] Section 3.2 says 'a set of 10 prompts' while Section 3.4 says 'a pool of 20 distinct prompts'. Please align these numbers.
  2. [Figure 1 and Eq. (2)] Figure 1's caption says hds and p are concatenated, but Eq. (2) includes the textual information z. The caption should be updated to reflect that z is also part of the input.
  3. [§1] The model name 'SALOMONN' appears in the introduction; it should be 'SALMONN' as elsewhere.
  4. [References] References [9] and [10] are duplicate entries for the WavLM paper. One should be removed.
  5. [§4.1] The distinction between EmoSLLM and EmoSLLM-base is introduced only in §4.1 but is used earlier (e.g., in the prompt setting paragraph). Clarify the naming and the exact contents of z (especially whether the transcript is included in z or provided separately at inference).

Circularity Check

0 steps flagged · score 2.0 of 10

No derivation-to-fit circularity; the main risk is an evaluation-protocol asymmetry (ground-truth transcript at inference), which is a benchmark-fairness concern, not a circular-derivation one.

full rationale

I found no circular derivation in the paper's core chain. The architecture is defined independently: audio features are h_AE = f_AE(x) (Eq. 1), the downsampling module produces h_ds = g(h_AE), and the LLM output is EmoSLLM(x,p,z) = f_LLM([h_ds,p,z]) (Eq. 2). Training minimizes the negative log-likelihood of the target label given these inputs (Eqs. 3-4), and the reported SER accuracy is measured on the held-out MSP-Podcast test1 split. No fitted parameter is renamed as a prediction, and no equation reduces the emotion prediction to the training target by construction. The self-citations (e.g., [28] and [55]) appear only in related-work surveys and are not load-bearing for the central claim. The notable issue is the inference protocol in Section 5.3 and Table 3: the main EmoSLLM variant receives the ground-truth transcript as a prefix ('| ASR: <transcript> | Emotion:') during generation, and the paper's own ablations show SER-only accuracy drops to 0.417 and Prompt-hint to 0.431. That is an evaluation-fairness / correctness concern about whether the comparison to audio-only baselines is apples-to-apples; it does not constitute derivation-to-fit circularity and therefore does not raise the circularity score beyond the minor-self-citation level.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The central claim rests on a standard set of pretrained components and hyperparameters. No new physical entities or forces are introduced. The most load-bearing assumptions are the transferability of QPMapper to audio and, critically, the availability of the ground-truth transcript at inference.

free parameters (6)
  • LoRA rank and alpha = r=8, alpha=16
    Chosen by hand to balance adaptation capacity with parameter efficiency; Section 4.1.
  • QPMapper number of queries = 32
    Determines compression of audio tokens; Section 4.1.
  • QPMapper transformer layers and attention heads = 2 layers, 8 heads
    Architecture capacity for the audio-to-LLM mapping; Section 4.1.
  • Learning rate and weight decay = 5e-4, 0.01
    Standard optimizer choices; Section 4.1.
  • Batch size = 512
    Set for training stability; Section 4.1.
  • Epochs and early stopping patience = 10/10/20, patience 2
    Training schedule and stopping criterion; Section 4.1.
assumptions (5)
  • domain assumption WavLM representations contain sufficient paralinguistic and linguistic information for SER when combined with the LLM.
    The method relies on the pretrained audio encoder to provide the only audio signal; Section 3.1.
  • domain assumption QPMapper can map audio features into the LLM's embedding space without degrading information.
    Adopted from vision work [57] and assumed transferable to audio; Section 3.1.
  • domain assumption Llama3.2-3B-Instruct can, via LoRA, learn to condition on audio tokens and textual hints for SER.
    Core to the method; Section 3.5.
  • domain assumption The ground-truth transcript is available and correct at inference.
    Inference relies on inserting the true transcript; Section 5.3.
  • domain assumption MSP-Podcast test1 is a valid and representative SER benchmark.
    Used as the sole evaluation set; Section 4.1.

how reviews work

0 comments
Cite this review

Pith. "Pith review of EmoSLLM: Parameter-Efficient Adaptation of LLMs for Speech Emotion Recognition." pith.science (2026). https://pith.science/paper/VQYXBENW

@misc{pith2026250814130,
  author       = {Pith},
  title        = {Pith review of: EmoSLLM: Parameter-Efficient Adaptation of LLMs for Speech Emotion Recognition},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VQYXBENW}},
  note         = {Machine review of arXiv:2508.14130}
}
read the original abstract

Emotion recognition from speech is a challenging task that requires capturing both linguistic and paralinguistic cues, with critical applications in human-computer interaction and mental health monitoring. Recent works have highlighted the ability of Large Language Models (LLMs) to perform tasks outside of the sole natural language area. In particular, recent approaches have investigated coupling LLMs with other data modalities by using pre-trained backbones and different fusion mechanisms. This work proposes a novel approach that fine-tunes an LLM with audio and text representations for emotion prediction. Our method first extracts audio features using an audio feature extractor, which are then mapped into the LLM's representation space via a learnable interfacing module. The LLM takes as input (1) the transformed audio features, (2) additional features in the form of natural language (e.g., the transcript), and (3) a textual prompt describing the emotion prediction task. To efficiently adapt the LLM to this multimodal task, we employ Low-Rank Adaptation (LoRA), enabling parameter-efficient fine-tuning. Experimental results on standard emotion recognition benchmarks demonstrate that our model outperforms all but one existing Speech-Text LLMs in the literature, while requiring less than half the parameters of competing approaches. This highlights our approach's effectiveness in integrating multi-modal inputs for speech-based emotion understanding while maintaining significant computational efficiency.

Figures

Figures reproduced from arXiv: 2508.14130 by the authors.

Figure 1
Figure 1. Em SLLM Pipeline. In step (a), the audio signal is fed to a pretrained audio encoder to obtain a vectorized embedded representation hAE of dimension R n×dAE . In step (b) a text prompt is fed to the embedding module of an LLM to output a vectorized embedded representation p of dimension R np×dLLM . In step (c) hAE is fed to a downsampling module and the obtained sequence hds is of dimension R nq×dLLM . In step (d) h… view at source ↗
Figure 2
Figure 2. Performance Comparison. Performance comparison with existing Audio-Language Models that perform speech emotion recognition, Qwen2-Audio-Instruct [12] (Qwen2-A), OASQA￾LLM [44] (OASQAL), and SIFT-LLM [44]. We compare Em SLLM with existing methods on MSP-Podcast test1 and observe that with fewer parameters and significantly less training time, our model displays strong SER performance as it outperforms all methods exc… view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. EmotionAI: A Privacy-Preserving Computational Intelligence Pipeline for Speech-Emotion-Grounded Conversational Analysis

    cs.SD 2026-06 unverdicted novelty 5.0 of 10

    A local pipeline makes speech-emotion evidence inspectable and answerable by a small LLM panel while staying on-device, but zero-shot emotion accuracy is poor and CPU runtime exceeds real time.

Reference graph

Works this paper leans on

74 extracted references · 59 canonical work pages · cited by 1 Pith paper

  1. [1]

    Alsaad, Yusra Azhar, and Yasser Al Nasser.Depression in Children

    Ali J. Alsaad, Yusra Azhar, and Yasser Al Nasser.Depression in Children. StatPearls Publishing, Treasure Island (FL), 2023

    work page 2023

  2. [2]

    Self-supervised learning from images with a joint- embedding predictive architecture

    Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Self-supervised learning from images with a joint- embedding predictive architecture. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15619–15629, 6 2023

    work page 2023

  3. [3]

    vq-wav2vec: Self-supervised learning of discrete speech representations

    Alexei Baevski, Steffen Schneider, and Michael Auli. vq-wav2vec: Self-supervised learning of discrete speech representations. In International Conference on Learning Representations, 2020

    work page 2020

  4. [4]

    wav2vec 2.0: A framework for self-supervised learning of speech representations

    Alexei Baevski, Henry Zhou, Abdel rahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations. ArXiv, abs/2006.11477, 2020

    arXiv 2006

  5. [5]

    VICReg: Variance-invariance-covariance regular- ization for self-supervised learning

    Adrien Bardes, Jean Ponce, and Yann LeCun. VICReg: Variance-invariance-covariance regular- ization for self-supervised learning. In International Conference on Learning Representations, 2022

    work page 2022

  6. [6]

    Revisiting feature prediction for learning visual representations from video

    Adrien Bardes, Quentin Garrido, Jean Ponce, Xinlei Chen, Michael Rabbat, Yann LeCun, Mido Assran, and Nicolas Ballas. Revisiting feature prediction for learning visual representations from video. Transactions on Machine Learning Research, 2024

    work page 2024

  7. [7]

    Florian Bordes, Richard Yuanzhe Pang, Anurag Ajay, Alexander C. Li, Adrien Bardes, Suzanne Petryk, Oscar Mañas, Zhiqiu Lin, Anas Mahmoud, Bargav Jayaraman, Mark Ibrahim, Melissa Hall, Yunyang Xiong, Jonathan Lebensold, Candace Ross, Srihari Jayakumar, Chuan Guo, Diane Bouchacourt, Haider Al-Tahan, Karthik Padthe, Vasu Sharma, Hu Xu, Xiaoqing Ellen Tan, Me...

    work page 2024

  8. [8]

    Unsupervised learning of visual features by contrasting cluster assignments

    Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in neural information processing systems, 33:9912–9924, 2020

    work page 2020

Show all 74 references
  1. [9]

    Wavlm: Large-scale self-supervised pre-training for full stack speech processing

    Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, Jian Wu, Long Zhou, Shuo Ren, Yanmin Qian, Yao Qian, Jian Wu, Michael Zeng, and Furu Wei. Wavlm: Large-scale self-supervised pre-training for full st...

    2021

  2. [10]

    Wavlm: Large-scale self-supervised pre-training for full stack speech processing

    Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, Jian Wu, Long Zhou, Shuo Ren, Yanmin Qian, Yao Qian, Micheal Zeng, and Furu Wei. Wavlm: Large-scale self-supervised pre-training for full stack speec...

    2021

  3. [11]

    BEATs: Audio pre-training with acoustic tokenizers

    Sanyuan Chen, Yu Wu, Chengyi Wang, Shujie Liu, Daniel Tompkins, Zhuo Chen, Wanxiang Che, Xiangzhan Yu, and Furu Wei. BEATs: Audio pre-training with acoustic tokenizers. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, e...

    2023

  4. [12]

    Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models

    Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou. Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models. arXiv preprint arXiv:2311.07919, 2023

    2023 arXiv

  5. [13]

    Qwen2-audio technical report, 2024

    Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen2-audio technical report, 2024

    2024

  6. [14]

    Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V

    Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinso...

    2022

  7. [15]

    An unsupervised autoregressive model for speech representation learning

    Yu-An Chung, Wei-Ning Hsu, Hao Tang, and James Glass. An unsupervised autoregressive model for speech representation learning. In Interspeech, 2019

    2019

  8. [16]

    Nilaksh Das, Saket Dingliwal, S. Ronanki, Rohit Paturi, David Huang, Prashant Mathur, Jie Yuan, Dhanush Bekal, Xing Niu, Sai Muralidhar Jayanthi, Xilai Li, Karel Mundnich, Monica Sunkara, Sundararajan Srinivasan, Kyu J Han, and Katrin Kirchhoff. Speechverse: A large-scale gene...

    2024 arXiv

  9. [17]

    Speech prosody in mental disorders

    Hongwei Ding and Yang Zhang. Speech prosody in mental disorders. Annual Review of Linguistics, 9(V olume 9, 2023):335–355, 2023. ISSN 2333-9691

    2023

  10. [18]

    Learning to predict activity progress by self-supervised video alignment

    Gerard Donahue and Ehsan Elhamifar. Learning to predict activity progress by self-supervised video alignment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 18667–18677, June 2024

    2024

  11. [19]

    Moshi: a speech-text foundation model for real-time dialogue, 2024

    Alexandre Défossez, Laurent Mazaré, Manu Orsini, Amélie Royer, Patrick Pérez, Hervé Jégou, Edouard Grave, and Neil Zeghidour. Moshi: a speech-text foundation model for real-time dialogue, 2024

    2024

  12. [20]

    Prompting large language models with speech recognition abilities

    Yassir Fathullah, Chunyang Wu, Egor Lakomkin, Junteng Jia, Yuan Shangguan, Ke Li, Jinxi Guo, Wenhan Xiong, Jay Mahadeokar, Ozlem Kalinli, Christian Fuegen, and Mike Seltzer. Prompting large language models with speech recognition abilities. In ICASSP 2024 - 2024 IEEE Internati...

    2024

  13. [21]

    Risk factors for problematic smartphone use in children and adolescents: a review of existing literature

    Linda Fischer-Grote, Oswald D Kothgassner, and Anna Felnhofer. Risk factors for problematic smartphone use in children and adolescents: a review of existing literature. neuropsychiatrie, 33(4):179, 2019

    2019

  14. [22]

    Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko

    Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent ...

    2020

  15. [23]

    A generalizable speech emotion recognition model reveals depression and remission

    Lasse Hansen, Yan-Ping Zhang, Detlef Wolf, Konstantinos Sechidis, Nicolai Ladegaard, and Riccardo Fusaroli. A generalizable speech emotion recognition model reveals depression and remission. Acta Psychiatrica Scandinavica, 145(2):186–199, 2022. 10

    2022

  16. [24]

    Hubert: Self-supervised speech representation learning by masked prediction of hidden units

    Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Trans. Audio, Speech and Lang. Proc., 29:3451–3460, October 2021

    2021

  17. [25]

    Robust wav2vec 2.0: Analyzing domain shift in self-supervised pre-training

    Wei-Ning Hsu, Anuroop Sriram, Alexei Baevski, Tatiana Likhomanenko, Qiantong Xu, Vineel Pratap, Jacob Kahn, Ann Lee, Ronan Collobert, Gabriel Synnaeve, and Michael Auli. Robust wav2vec 2.0: Analyzing domain shift in self-supervised pre-training. In Interspeech 2021, pages 721–...

    2021

  18. [26]

    LoRA: Low-rank adaptation of large language models

    Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022

    2022

  19. [27]

    WavLLM: Towards robust and adaptive speech large language model

    Shujie Hu, Long Zhou, Shujie Liu, Sanyuan Chen, Lingwei Meng, Hongkun Hao, Jing Pan, Xunying Liu, Jinyu Li, Sunit Sivasankaran, Linquan Liu, and Furu Wei. WavLLM: Towards robust and adaptive speech large language model. In Findings of the Association for Computa- tional Lingui...

    2024

  20. [28]

    Ibrahim, Antony Perzo, and Simon Leglaive

    Karim M. Ibrahim, Antony Perzo, and Simon Leglaive. Towards improving speech emotion recognition using synthetic data augmentation from emotion conversion. In ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 10636–10640, 2024

    2024

  21. [29]

    Audio flamingo: A novel audio language model with few-shot learning and dialogue abilities

    Zhifeng Kong, Arushi Goel, Rohan Badlani, Wei Ping, Rafael Valle, and Bryan Catanzaro. Audio flamingo: A novel audio language model with few-shot learning and dialogue abilities. In Forty-first International Conference on Machine Learning, 2024

    2024

  22. [30]

    Masked vision and language modeling for multi-modal representation learning

    Gukyeong Kwon, Zhaowei Cai, Avinash Ravichandran, Erhan Bas, Rahul Bhotika, and Stefano Soatto. Masked vision and language modeling for multi-modal representation learning. In The Eleventh International Conference on Learning Representations, 2023

    2023

  23. [31]

    Modeling caption diversity in contrastive vision-language pretraining

    Samuel Lavoie, Polina Kirichenko, Mark Ibrahim, Mido Assran, Andrew Gordon Wilson, Aaron Courville, and Nicolas Ballas. Modeling caption diversity in contrastive vision-language pretraining. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, J...

    2024

  24. [32]

    Binning as a pretext task: Improving self-supervised learning in tabular domains

    Kyungeun Lee, Ye Seul Sim, Hyeseung Cho, Moonjung Eo, Suhee Yoon, Sanghyu Yoon, and Woohyung Lim. Binning as a pretext task: Improving self-supervised learning in tabular domains. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceeding...

    2024

  25. [33]

    Blip-2: bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In Proceedings of the 40th International Conference on Machine Learning, ICML’23. JMLR.org, 2023

    2023

  26. [34]

    Revise, reason, and recognize: Llm-based emotion recognition via emotion-specific prompts and asr error correction

    Yuanchao Li, Yuan Gong, Chao-Han Huck Yang, Peter Bell, and Catherine Lai. Revise, reason, and recognize: Llm-based emotion recognition via emotion-specific prompts and asr error correction. In ICASSP 2025 - 2025 IEEE International Conference on Acoustics, Speech and Signal Pr...

    2025

  27. [35]

    Liu, Yu-An Chung, and James R

    Alexander H. Liu, Yu-An Chung, and James R. Glass. Non-autoregressive predictive coding for learning speech representations from local dependencies. In Interspeech, 2020

    2020

  28. [36]

    Liu, Shang-Wen Li, and Hung yi Lee

    Andy T. Liu, Shang-Wen Li, and Hung yi Lee. Tera: Self-supervised learning of transformer encoder representation for speech. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:2351–2366, 2020. 11

    2020

  29. [37]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019

    2019

  30. [38]

    Building naturalistic emotionally balanced speech corpus by retrieving emotional speech from existing podcast recordings

    Reza Lotfian and Carlos Busso. Building naturalistic emotionally balanced speech corpus by retrieving emotional speech from existing podcast recordings. IEEE Transactions on Affective Computing, 10(4):471–483, 2019

    2019

  31. [39]

    The llama 3 herd of models, 2024

    AI @Meta. The llama 3 herd of models, 2024

    2024

  32. [40]

    Byol for audio: Self-supervised learning for general-purpose audio representation

    Daisuke Niizumi, Daiki Takeuchi, Yasunori Ohishi, Noboru Harada, and Kunio Kashino. Byol for audio: Self-supervised learning for general-purpose audio representation. In 2021 International Joint Conference on Neural Networks (IJCNN), pages 1–8, 2021

    2021

  33. [41]

    Byol for audio: Exploring pre-trained general-purpose audio representations

    Daisuke Niizumi, Daiki Takeuchi, Yasunori Ohishi, Noboru Harada, and Kunio Kashino. Byol for audio: Exploring pre-trained general-purpose audio representations. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 31:137–151, 2023

    2023

  34. [42]

    Olson, Dasha A

    Jay A. Olson, Dasha A. Sandra, Élissa S. Colucci, Alain Al Bikaii, Denis Chmoulevitch, Johnny Nahas, Amir Raz, and Samuel P.L. Veissière. Smartphone addiction is increasing across the world: A meta-analysis of 24 countries. Computers in Human Behavior, 129:107138, 2022. ISSN 0747-5632

    2022

  35. [43]

    Librispeech: An asr corpus based on public domain audio books

    Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: An asr corpus based on public domain audio books. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5206–5210, 2015

    2015

  36. [44]

    Strimel, and Andreas Schwarz

    Prabhat Pandey, Rupak Vignesh Swaminathan, K V Vijay Girish, Arunasish Sen, Jian Xie, Grant P. Strimel, and Andreas Schwarz. Sift-50m: A large-scale multilingual dataset for speech instruction fine-tuning, 2025

    2025

  37. [45]

    Emotion Recognition from Speech Using wav2vec 2.0 Embeddings

    Leonardo Pepino, Pablo Riera, and Luciana Ferrer. Emotion Recognition from Speech Using wav2vec 2.0 Embeddings. In Interspeech, pages 3400–3404, 2021

    2021

  38. [46]

    V oice modulation: A window into the origins of human vocal control? Trends in Cognitive Sciences, 20(4):304–318, 2016

    Katarzyna Pisanski, Valentina Cartei, Carolyn McGettigan, Jordan Raine, and David Reby. V oice modulation: A window into the origins of human vocal control? Trends in Cognitive Sciences, 20(4):304–318, 2016. ISSN 1364-6613

    2016

  39. [47]

    Qwen2.5 technical report, 2025

    Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, ...

    2025

  40. [48]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila an...

    2021

  41. [49]

    Robust speech recognition via large-scale weak supervision

    Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine Mcleavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In Andreas Krause, Emma Brun- skill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Pro- ce...

    2023

  42. [50]

    Generating diverse high-fidelity images with vq-vae-2

    Ali Razavi, Aaron van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d 'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associ...

    2019

  43. [51]

    wav2vec: Unsupervised pre-training for speech recognition

    Steffen Schneider, Alexei Baevski, Ronan Collobert, and Michael Auli. wav2vec: Unsupervised pre-training for speech recognition. CoRR, abs/1904.05862, 2019

    1904 arXiv

  44. [52]

    Flava: A foundational language and vision alignment model

    Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojciech Galuba, Marcus Rohrbach, and Douwe Kiela. Flava: A foundational language and vision alignment model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion (CVPR), pages...

    2022

  45. [53]

    SALMONN: Towards generic hearing abilities for large language models

    Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun MA, and Chao Zhang. SALMONN: Towards generic hearing abilities for large language models. In The Twelfth International Conference on Learning Representations, 2024

    2024

  46. [54]

    Chameleon: Mixed-modal early-fusion foundation models, 2024

    Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models, 2024

    2024

  47. [55]

    T-JEPA: Augmentation-free self-supervised learning for tabular data

    Hugo Thimonier, José Lucas De Melo Costa, Fabrice Popineau, Arpad Rimmel, and Bich-Liên Doan. T-JEPA: Augmentation-free self-supervised learning for tabular data. In The Thirteenth International Conference on Learning Representations, 2025

    2025

  48. [56]

    Maria Tsimpoukelli, Jacob Menick, Serkan Cabi, S. M. Ali Eslami, Oriol Vinyals, and Felix Hill. Multimodal few-shot learning with frozen language models. In A. Beygelzimer, Y . Dauphin, P. Liang, and J. Wortman Vaughan, editors,Advances in Neural Information Processing Systems, 2021

    2021

  49. [57]

    Improved baselines for data-efficient perceptual augmentation of LLMs, 2024

    Théophane Vallaeys, Mustafa Shukor, Matthieu Cord, and Jakob Verbeek. Improved baselines for data-efficient perceptual augmentation of LLMs, 2024

    2024

  50. [58]

    Neural discrete representation learning

    Aaron van den Oord, Oriol Vinyals, and koray kavukcuoglu. Neural discrete representation learning. In I. Guyon, U. V on Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Ass...

    2017

  51. [59]

    Schuller

    Johannes Wagner, Andreas Triantafyllopoulos, Hagen Wierstorf, Maximilian Schmitt, Felix Burkhardt, Florian Eyben, and Björn W. Schuller. Dawn of the transformer era in speech emotion recognition: Closing the valence gap. IEEE Transactions on Pattern Analysis and Machine Intell...

    2023

  52. [60]

    A fine-tuned wav2vec 2.0/hubert benchmark for speech emotion recognition, speaker verification and spoken language understanding

    Yingzhi Wang, Abdelmoumene Boumadane, and Abdelwahab Heba. A fine-tuned wav2vec 2.0/hubert benchmark for speech emotion recognition, speaker verification and spoken language understanding. CoRR, abs/2111.02735, 2021

    2021 arXiv

  53. [61]

    Towards latent masked image modeling for self-supervised visual representation learning

    Yibing Wei, Abhinav Gupta, and Pedro Morgado. Towards latent masked image modeling for self-supervised visual representation learning. In Aleš Leonardis, Elisa Ricci, Stefan Roth, Olga Russakovsky, Torsten Sattler, and Gül Varol, editors,Computer Vision – ECCV 2024, pages 1–17, 2025

    2024

  54. [62]

    On decoder-only architecture for speech-to-text and large language model integration, 2023

    Jian Wu, Yashesh Gaur, Zhuo Chen, Long Zhou, Yimeng Zhu, Tianrui Wang, Jinyu Li, Shujie Liu, Bo Ren, Linquan Liu, and Yu Wu. On decoder-only architecture for speech-to-text and large language model integration, 2023

    2023

  55. [63]

    Switchtab: Switched autoencoders are effective tabular learners

    Jing Wu, Suiyao Chen, Qi Zhao, Renat Sergazinov, Chen Li, Shengjie Liu, Chongchao Zhao, Tianpei Xie, Hanqing Guo, Cheng Ji, Daniel Cociorva, and Hakan Brunzell. Switchtab: Switched autoencoders are effective tabular learners. Proceedings of the AAAI Conference on Artificial In...

    2024

  56. [64]

    Qwen3 technical report

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jia...

    2025 arXiv

  57. [65]

    Detection of mood disorder using speech emotion profiles and lstm

    Tsung-Hsien Yang, Chung-Hsien Wu, Kun-Yi Huang, and Ming-Hsiang Su. Detection of mood disorder using speech emotion profiles and lstm. In 2016 10th International Symposium on Chinese Spoken Language Processing (ISCSLP), pages 1–5, 2016

    2016

  58. [66]

    Coca: Contrastive captioners are image-text foundation models

    Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. Transactions on Machine Learning Research, 2022

    2022

  59. [67]

    Connecting speech encoder and large language model for asr

    Wenyi Yu, Changli Tang, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, and Chao Zhang. Connecting speech encoder and large language model for asr. InICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 126...

    2024

  60. [68]

    Barlow twins: Self- supervised learning via redundancy reduction

    Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self- supervised learning via redundancy reduction. In International conference on machine learning, pages 12310–12320. PMLR, 2021

    2021

  61. [69]

    XTab: cross-table pretraining for tabular transformers

    Bingzhao Zhu, Xingjian Shi, Nick Erickson, Mu Li, George Karypis, and Mahsa Shoaran. XTab: cross-table pretraining for tabular transformers. In Proceedings of the 40th International Conference on Machine Learning, ICML’23, 2023

    2023

  62. [70]

    role": "system

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhanc- ing vision-language understanding with advanced large language models, 2023. 14 A Prompts We provide in this section a more comprehensive description of the prompt structures used to train ...

    2023

  63. [71]

    **Automatic Speech Recognition (ASR)** — transcribe the spoken content

  64. [72]

    role": "user

    **Speech Emotion Recognition (SER)** — identify the emotion expressed in the audio. Follow one of the two output formats: - For ASR-only tasks: ’| ASR: <transcription> |’ - For SER-only tasks: ’| Emotion: <emotion code> |’ For tasks involving both ASR and SER, use the followin...

  65. [73]

    Identify and write down the words spoken in the preceding audio

    Task: **Automatic Speech Recognition (ASR)**. Identify and write down the words spoken in the preceding audio

  66. [74]

    role": "user

    Task: **Speech Emotion Recognition**. Analyze the audio and determine the emotional state of the speaker." } A.5 Supplementary features hinting We provide a variety of supplementary features to guide the LLM in its prediction. We include features ranging from the gender of the...

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.