Pith. sign in

REVIEW 4 major objections 4 minor 56 references

Towards LLM-Empowered Fine-Grained Speech Descriptors for Explainable Emotion Recognition

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read An end-to-end LLM-based system that disentangles pitch, tone, and emphasis from HuBERT features and predicts them alongside emotion labels achieves 77.6% unweighted accuracy on IEMOCAP and 60.1% on MELD, outperforming comparable…

desk verdict A plausible end-to-end SER+descriptor pipeline with consistent gains, but the statistical significance claim is unsupported as written; worth reviewing after the stats and error bars are fixed. read the letter →

arxiv 2505.23236 v1 pith:27GZNVEC submitted 2025-05-29 cs.SD cs.HCeess.AS

Youjun Chen , Xurong Xie , Haoning Xu , Mengzhe Geng , Guinan Li , Chengxi Deng , Huimeng Wang , Shujie Hu
show 1 more author
Xunying Liu
This is my paper
classification cs.SDcs.HCeess.AS
keywords speechemotionrecognitiondescriptorslargelanguagemodelsvariationalautoencoderinformationbottleneckfeaturedisentanglementexplainability
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to show that speech emotion recognition (SER) improves, and becomes explainable, when a large language model is trained not just to predict a single emotion label but to first separate content-related from emotion-descriptor-related information—pitch, tone, emphasis—inside a frozen self-supervised speech representation. The proposed architecture is end-to-end: raw audio goes into HuBERT, is compressed through an information-bottleneck VAE-style block, and is decoded by Llama-3.1 to produce a transcript, emotion descriptors, and an emotion label in one pass. On IEMOCAP and MELD, the authors report unweighted accuracy of 77.6% and 60.1%, beating their LLaMA-based baselines by up to 4.0% and 3.7% absolute, and they claim these gains are statistically significant under a paired t-test. If true, this matters because the predicted emotion descriptors give a human-readable rationale for each emotion label, and removing the external ASR dependency means classification and explanation come from the same model.

What carries the argument

The load-bearing mechanism is alternating multi-task fine-tuning over two stages: first the content encoder and adapter are trained on the ASR transcription task while descriptor tokens are held at zero; then the content path is frozen and the descriptor encoder and adapter are trained on the joint SER-SED and ASR objective. The two encoders read all hidden layers of a frozen HuBERT-large encoder, form a weighted sum, and pass through a variational information bottleneck (VIB) block that compresses the representation $Z$ while keeping task information, with loss $L_{\mathrm{VIB}} = \mathbb{E}[-\log q(y|z)] + \beta\, \mathrm{KL}(p(z|x), p_0(z))$. The compressed content and descriptor embeddings are mapped by adapters into the embedding space of Llama-3.1-8B-Instruct, whose autoregressive output is the transcript, descriptor, and emotion label. LoRA provides parameter-efficient fine-tuning; $\beta$ controls the compression-prediction trade-off, values in $[10^{-3}, 10^{-1}]$ keep SER stable, and $\beta=1$ degenerates to a standard VAE.

What would settle it

Recompute the IEMOCAP and MELD comparisons with a fully specified paired significance procedure: define the pairing units (session, speaker, or utterance), the independence assumptions, and a bootstrap or permutation distribution, and check whether the 4.0% and 3.7% absolute accuracy gaps remain significant; also inspect the cited reference to see if it actually provides the test. If the 95% interval for the mean difference includes zero or the reference does not describe a paired single-tailed t-test, the significance claim fails.

Watch

Extended reading notes

Core claim

The central claim is that jointly training an LLM decoder to recognize emotion while also generating fine-grained speech emotion descriptors—pitch, tone, emphasis—and a transcript from a single speech representation improves both tasks. The paper argues that the descriptor path is not a side output but an active contributor: on IEMOCAP, adding descriptor prediction to a content-only disentangled system raises unweighted accuracy by 7.0% absolute, and the full system reaches 77.6% on IEMOCAP and 60.1% on MELD, with word error rates of 11.1% and 25.3%. The authors attribute the gains to separating content-related and descriptor-related information through an information-bottleneck VAE and to alternating fine-tuning on ASR and SER-SED tasks, which they show outperforms one-stage, two-stage, and non-alternating training. They also report that the generated descriptors score above published SED captioning methods on EMOSEC under zero-shot evaluation.

Load-bearing premise

The load-bearing premise is that the paired t-test used to claim significance is valid; the paper never specifies what is paired, how many independent observations there are, or why the cited test applies, and MELD's single split leaves no obvious within-dataset pairing, so if that test is not valid the 'statistically significant' part of the claim is unsupported.

Editorial extensions

If this is right

  • SER can be made end-to-end: transcript, emotion descriptors, and emotion labels are produced from raw speech in a single model, so performance no longer depends on an external ASR's errors.
  • Descriptor prediction is quantitatively coupled to SER accuracy: removing it costs 7.0% unweighted accuracy on IEMOCAP and 6.8% on MELD, so explainability is not a free add-on but part of what drives classification.
  • Information-bottleneck compression is useful for feature disentanglement in SER: standard VAE behavior ($\beta=1$) degrades accuracy, while $\beta$ between $10^{-3}$ and $10^{-1}$ is stable.
  • Alternating fine-tuning avoids the overfitting that prompt-repetitive LoRA fine-tuning causes, improving both SED and SER over one-stage or two-stage sequential training.
  • The generated descriptors are competitive with published speech-emotion captioning systems on EMOSEC under zero-shot evaluation, including BLEU@4 of 10.3 and CIDEr of 19.3.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial — the alternating content/descriptor separation could transfer to other SSL speech encoders and to related tasks needing interpretable intermediate labels, such as prosody or speaker-state analysis.
  • Editorial — because training data are dominated by Mandarin emotion speech, the size of the English benchmark gains may depend on language and emotion distribution; a balanced multilingual SED training corpus would show how much transfers.
  • Editorial — a practical byproduct is that the predicted descriptors could serve as a human-auditable explanation interface for SER, letting users verify why an emotion label was chosen.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes an end-to-end LLM-empowered speech emotion recognition (SER) system that combines HuBERT representations, information-bottleneck-based feature disentanglement, and alternating multi-task fine-tuning of a LLaMA decoder on ASR and joint SER-SED tasks. The authors claim statistically significant SER unweighted accuracy improvements of up to 4.0% absolute on IEMOCAP and 3.7% absolute on MELD over a comparable LLaMA-based baseline, and further claim that predicted fine-grained speech emotion descriptors provide explainability for SER. Experiments are reported on EMOSEC for SED prediction and on IEMOCAP/MELD for SER, with ablations over the IB weight, training strategy, and target outputs.

Significance. If the central claims hold, the paper offers a useful architectural contribution: it is, to my knowledge, the first to combine feature disentanglement with fine-grained speech emotion descriptor generation in an end-to-end LLM-based SER system, and it releases code. The consistent positive ordering of the proposed system across Tables 1-3 is encouraging, and the ablation study is a genuine experimental effort. However, the two headline claims—statistical significance and explainability—are not currently supported by the evidence presented: the significance test is unspecified and cited to an unrelated paper, and no quantitative or human evaluation of explainability is provided. The paper would be strengthened substantially by adding a proper significance analysis (or removing the significance wording) and by validating the explainability claim with external judgments or a measurable proxy.

major comments (4)
  1. [Abstract; Sec. 4.3; Tables 1-3] The central claim of 'statistically significant' improvements is not established. The only description of the test is 'Paired Single-tailed t TEST [44], p=0.05', but reference [44] is a paper on bilingual emotion perception, not a statistical methods reference, and no test statistic, degrees of freedom, per-fold/per-run accuracies, or exact p-values are reported anywhere. For IEMOCAP, the natural pairing unit is the leave-one-session-out fold (n=5), which is too small for a valid t-test given the heavy training-set overlap; for MELD, there is a single official split, so no within-dataset paired test is possible unless repeated training runs or utterance-level resampling are described, and none are. The reported improvements of 4.0% and 3.7% absolute therefore cannot currently be distinguished from run-to-run variability.
  2. [Sec. 4.5, Table 3] The text states that 'removing any of these techniques, or modifying the parameter settings all lead to performance degradation,' but Table 3 shows that Sys. 7 (β=1e−3) achieves MELD UA of 60.2% and B@4 of 10.4, both slightly higher than the corresponding values for Sys. 4 (60.1% and 10.3). If these differences are within noise, the claim of universal degradation requires error bars or a significance test; if they are real, the claim is factually incorrect. This weakens the ablation-based justification for the chosen β.
  3. [Abstract; Sec. 1; Sec. 4.5] The explainability claim is not measured or validated. The predicted emotion descriptors are generated by the same model whose emotion labels they are said to explain, and no human evaluation, external SED grounding, or quantitative explainability metric is provided. The improvements in SER accuracy (e.g., Sys. 4 vs. Sys. 3 in Table 3) show a correlation between SED prediction and SER, but they do not demonstrate that the descriptors explain the predictions in any human-understandable or verifiable sense. The paper should either add an explainability evaluation or temper the claim to 'descriptor generation improves SER accuracy.'
  4. [Sec. 4.4; Eq. (2)] The description of the training details is too sparse to assess reproducibility of the statistical claims. In particular, the number of training runs, random seeds, and variance across runs are not reported, and the IB regularization weight β is described as 'iterat[ing] from 1e0 to 1e−4' without specifying the schedule (e.g., linear, exponential, or stepwise). This matters because the headline results are single numbers with no associated uncertainty, and the significance claim depends on run-to-run variability.
minor comments (4)
  1. [Sec. 4.5, Table 1 caption] The relative improvement '80.4&' appears to be a typo for '80.4%'.
  2. [Sec. 2.2 and throughout] There are inconsistent spacing and formatting issues, e.g., 'V AE' and 'Eq. (2)' should be 'VAE' and 'Eq. (2)' respectively.
  3. [Reference [44]] Reference [44] is cited as the source of the paired t-test procedure, but the reference is a bilingual emotion perception study; this citation appears to be erroneous and should be replaced with a proper statistical methods reference or the test description should be self-contained.
  4. [Sec. 4.5, Table 3 caption and text] The text identifies 'Sys. 4' as the best system and notes it is the same as Sys. 3 in Table 1 and Sys. 7 in Table 2; this cross-referencing is clear, but the numbering of systems across tables is confusing and should be unified or explicitly mapped.

Circularity Check

0 steps flagged · score 1.0 of 10

No circular derivation: the empirical SER/SED comparisons are self-contained; the mild self-referential explainability framing and an unsupported significance-test citation are weaknesses, not circularity.

full rationale

The paper's central quantitative claims are empirical comparisons (Sys. 7 vs. Sys. 4-6 on IEMOCAP/MELD, and SED prediction on EMOSEC), not analytical derivations from fitted values or from prior results by the same authors. The IB/VAE background cites the authors' own prior work ([31], [32]) and standard references ([28]-[30]), but these citations are not load-bearing: the reported accuracy figures are produced by running the described architecture and fine-tuning, and the ablations in Tables 1-3 are legitimate experiments. The 'statistically significant' language is unsupported because the cited test reference [44] is unrelated to statistical methodology and no pairing units, degrees of freedom, or p-values are reported (Sec. 4.3, Tables 1-3), but this is a verification/correctness weakness, not a circular step. The 'explainability' claim that predicted SED tokens 'offer further explainability for SER' is self-referential in that the descriptors are generated by the same model whose SER output they are said to explain, and no external user study validates the explanation; however, this affects the interpretive framing, not the numeric results. No equation in the paper equates a prediction to its input by construction, and no self-citation chain forces the reported outcomes. Consistent with the default expectation for empirical papers, the derivation chain is self-contained and no significant circularity is present.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The paper relies on standard SSL and LLM fine-tuning assumptions plus the validity of its statistical evaluation. The main free parameter is beta, tuned on validation; several training hyperparameters are unstated.

free parameters (3)
  • IB regularization weight beta = e^{-2}
    Selected from {1, e^{-1}, e^{-2}, e^{-3}, e^{-4}} as the best-performing setting in Table 3; controls compression vs. prediction trade-off.
  • Stage-2 peak learning rate = 2e-5
    Hyperparameter reported in Sec. 4.4; chosen by the authors but not argued for.
  • LoRA rank
    Not reported in the paper; a capacity hyperparameter that affects fine-tuning behavior and is not documented.
assumptions (5)
  • domain assumption Frozen HuBERT-large representations contain sufficient task-relevant information for SER, SED, and ASR.
    Sec. 3.1 uses all HuBERT hidden-layer outputs as X; if this fails, the whole architecture loses its input signal.
  • standard math The variational information bottleneck loss (Eq. 2) is a valid and tractable surrogate for the IB objective.
    Sec. 2.2 invokes standard VIB bounds and the reparameterization trick; this is textbook.
  • domain assumption LoRA fine-tuning with the given prompt template enables the LLaMA-3.1-8B model to perform the three target tasks without catastrophic forgetting.
    Sec. 2.1 and 3.4 rely on the LM following instructions in the prompt; no instruction-following evaluation is provided.
  • domain assumption SpeechCraft descriptor annotations are reliable training targets and EMOSEC captions are a valid zero-shot evaluation set.
    Sec. 4.1 selects these datasets; the paper does not audit annotation quality.
  • domain assumption The paired single-tailed t-test (p=0.05) is valid for the reported comparisons.
    Sec. 4.3 states significance via [44], but the test procedure is unspecified and the citation is unrelated; the assumption is questionable.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Towards LLM-Empowered Fine-Grained Speech Descriptors for Explainable Emotion Recognition." pith.science (2026). https://pith.science/paper/27GZNVEC

@misc{pith2026250523236,
  author       = {Pith},
  title        = {Pith review of: Towards LLM-Empowered Fine-Grained Speech Descriptors for Explainable Emotion Recognition},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/27GZNVEC}},
  note         = {Machine review of arXiv:2505.23236}
}
read the original abstract

This paper presents a novel end-to-end LLM-empowered explainable speech emotion recognition (SER) approach. Fine-grained speech emotion descriptor (SED) features, e.g., pitch, tone and emphasis, are disentangled from HuBERT SSL representations via alternating LLM fine-tuning to joint SER-SED prediction and ASR tasks. VAE compressed HuBERT features derived via Information Bottleneck (IB) are used to adjust feature granularity. Experiments on the IEMOCAP and MELD benchmarks demonstrate that our approach consistently outperforms comparable LLaMA-based SER baselines, including those using either (a) alternating multi-task fine-tuning alone or (b) feature disentanglement only. Statistically significant increase of SER unweighted accuracy by up to 4.0% and 3.7% absolute (5.4% and 6.6% relative) are obtained. More importantly, emotion descriptors offer further explainability for SER.

Figures

Figures reproduced from arXiv: 2505.23236 by the authors.

Figure 1
Figure 1. Illustration of the proposed LLM-empowered explainable speech emotion recognition approach and prompt template. tive research on the importance of fine-grained SED predic￾tion for SER performance [5, 23–26]. Our code is avail￾able at https://github.com/SEUJames23/explainable-emotion￾recognition. 2. Feature Disentanglement 2.1. Alternating Multi-task Fine-tuning A parameter-efficient fine-tuning method known as low-r… view at source ↗
Figure 2
Figure 2. Illustration of two stages of the whole training pro￾cess on the two sequential downstream tasks with the alternat￾ing multi-task fine-tuning strategy. 4.2. Baseline Systems We compare the proposed method with the following baselines: HuBERT+LLM-SER+SED+ASR: LLaMA SER system fine￾tuned to joint SER-SED prediction and ASR tasks using stan￾dard HuBERT features. HuBERT+Feature Disentanglement(FD)+LLM-SER+ASR: LLaMA SER… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

56 extracted references · 56 canonical work pages

  1. [44]

    Distilhubert: Speech representation learning by layer-wise distillation of hidden-unit bert,

    H.-J. Chang et al., “Distilhubert: Speech representation learning by layer-wise distillation of hidden-unit bert,” in ICASSP, 2022

    work page 2022

  2. [1]

    Despite decades of promising advance- ments, most research [1–4] has primarily focused on classify- ing speech into single, discrete emotion categories

    Introduction Speech emotion recognition (SER) is fundamental to human- computer interaction. Despite decades of promising advance- ments, most research [1–4] has primarily focused on classify- ing speech into single, discrete emotion categories. However, such approaches oversimplify the complexity of human emo- tions, as single-label outputs fail to captu...

    work page

  3. [2]

    In contrast, previous research paid attention to either feature disentan- glement [6–17] or fine-grained emotion descriptor prediction [5, 23–26] alone

    To the best of our knowledge, this paper presents the first work to use both feature disentanglement and fine-grained speech emotion descriptors for explainable SER. In contrast, previous research paid attention to either feature disentan- glement [6–17] or fine-grained emotion descriptor prediction [5, 23–26] alone

    work page

  4. [3]

    In contrast, prior research employs external ASR models to generate transcripts before performing SER [12–17], which may lead to low SER performance affected by ASR er- rors

    This paper introduces a novel end-to-end LLM-based ex- plainable SER architecture that simultaneously generates tran- script, emotion descriptors and emotion labels directly from in- put speech. In contrast, prior research employs external ASR models to generate transcripts before performing SER [12–17], which may lead to low SER performance affected by A...

    work page

  5. [4]

    This paper investigates the tight connection between fine-grained SED prediction and SER performance. In con- trast, prior research focused on generating fine-grained and precise emotion descriptors alone without further quantita- arXiv:2505.23236v1 [cs.SD] 29 May 2025 Figure 1: Illustration of the proposed LLM-empowered explainable speech emotion recogni...

    work page Pith review arXiv 2025

  6. [5]

    Alternating Multi-task Fine-tuning A parameter-efficient fine-tuning method known as low-rank adaptation, or LoRA [27], is widely employed to fine-tune the LLM-based models

    Feature Disentanglement 2.1. Alternating Multi-task Fine-tuning A parameter-efficient fine-tuning method known as low-rank adaptation, or LoRA [27], is widely employed to fine-tune the LLM-based models. However, extensively fine-tuning the model using the LoRA method on a large amount of prompt- repetitive speech-text data may cause the model to overfit o...

    work page

  7. [6]

    14200220, 14200021, 14200324, Innovation Technology Fund grant No

    Acknowledgements This research is supported by Hong Kong RGC GRF grant No. 14200220, 14200021, 14200324, Innovation Technology Fund grant No. ITS/218/21, the project of China Disabled Persons Federation No. CDPF2023KF00002, Youth Innovation Promo- tion Association CAS grant No. 2023119, and the project of Guangzhou Civil Affairs Science and Technology Fou...

    work page

  8. [7]

    1, the architecture of our approach consists of a HuBERT encoder, two feature disentanglement blocks, two feature adapters and an LLM decoder

    System Architecture As is illustrated in Fig. 1, the architecture of our approach consists of a HuBERT encoder, two feature disentanglement blocks, two feature adapters and an LLM decoder. Here, we de- note the HuBERT representations, intermediate representations, and target outputs as X, Z (ZCon for content-related, ZDes for emotion descriptor-related), ...

    work page

Show all 56 references
  1. [8]

    excited” with “happy

    Experiment 4.1. Datasets We use publicly available SED prediction and SER datasets in the training and evaluation stages. Training. SpeechCraft [37] annotates the utterances from four well-known corpora in a unified and fine-grained manner, which mainly considers the emotion d...

  2. [9]

    Fine-grained SED features are dis- entangled from HuBERT SSL representations via alternating LLM fine-tuning to joint SER-SED prediction and ASR tasks

    Conclusion This paper presents a novel end-to-end LLM-empowered ex- plainable SER approach. Fine-grained SED features are dis- entangled from HuBERT SSL representations via alternating LLM fine-tuning to joint SER-SED prediction and ASR tasks. V AE compressed HuBERT features d...

  3. [10]

    Leveraging speech ptm, text llm, and emotional tts for speech emotion recognition,

    Z. Ma et al., “Leveraging speech ptm, text llm, and emotional tts for speech emotion recognition,” in ICASSP, 2024

    2024

  4. [11]

    Emotion neural transducer for fine-grained speech emotion recognition,

    S. Shen et al., “Emotion neural transducer for fine-grained speech emotion recognition,” in ICASSP, 2024

    2024

  5. [12]

    Generalization of self-supervised learning- based representations for cross-domain speech emotion recogni- tion,

    A. R. Naini et al., “Generalization of self-supervised learning- based representations for cross-domain speech emotion recogni- tion,” in ICASSP, 2024

    2024

  6. [13]

    Noise-robust speech emotion recognition us- ing shared self-supervised representations with integrated speech enhancement,

    J.-T. Tzeng et al., “Noise-robust speech emotion recognition us- ing shared self-supervised representations with integrated speech enhancement,” in ICASSP, 2025

    2025

  7. [14]

    Emix: a data augmentation method for speech emotion recognition,

    A. Dang et al., “Emix: a data augmentation method for speech emotion recognition,” in ICASSP, 2023

    2023

  8. [15]

    Secap: Speech emotion captioning with large lan- guage model,

    Y . Xu et al., “Secap: Speech emotion captioning with large lan- guage model,” in AAAI, 2024

    2024

  9. [17]

    Disentangling textual and acoustic features of neural speech representations,

    H. Mohebbi et al., “Disentangling textual and acoustic features of neural speech representations,” arXiv preprint, 2024

    2024

  10. [18]

    Variational information bottleneck for effective low- resource audio classification,

    S. Si et al., “Variational information bottleneck for effective low- resource audio classification,” INTERSPEECH, 2021

    2021

  11. [19]

    Multi-modal emotion recognition using multiple acoustic features and dual cross-modal transformer,

    Y . Wu et al., “Multi-modal emotion recognition using multiple acoustic features and dual cross-modal transformer,” in ICASSP, 2024

    2024

  12. [20]

    Wavllm: Towards robust and adaptive speech large language model,

    S. Hu et al., “Wavllm: Towards robust and adaptive speech large language model,” EMNLP, 2024

    2024

  13. [21]

    Variational information bottleneck based regular- ization for speaker recognition

    D. Wang et al., “Variational information bottleneck based regular- ization for speaker recognition.” in INTERSPEECH, 2021

    2021

  14. [22]

    Speech emotion recognition combining acous- tic features and linguistic information in a hybrid support vector machine-belief network architecture,

    B. Schuller et al., “Speech emotion recognition combining acous- tic features and linguistic information in a hybrid support vector machine-belief network architecture,” in ICASSP, 2004

    2004

  15. [23]

    Speech emotion recognition with multi-level acoustic and semantic information extraction and interaction,

    Y . Gao et al., “Speech emotion recognition with multi-level acoustic and semantic information extraction and interaction,” in INTERSPEECH, 2024

    2024

  16. [25]

    Multimodal emotion recognition with high-level speech and text features,

    M. R. Makiuchi et al., “Multimodal emotion recognition with high-level speech and text features,” inASRU, 2021

    2021

  17. [26]

    Frontend attributes disentanglement for speech emotion recognition,

    Y .-X. Xi et al., “Frontend attributes disentanglement for speech emotion recognition,” in ICASSP, 2022

    2022

  18. [27]

    Speech emotion recognition with acoustic and lexi- cal features,

    Q. Jin et al., “Speech emotion recognition with acoustic and lexi- cal features,” in ICASSP, 2015

    2015

  19. [28]

    Salmonn: Towards generic hearing abilities for large language models,

    C. Tang et al., “Salmonn: Towards generic hearing abilities for large language models,” arXiv preprint, 2023

    2023

  20. [29]

    has been introduced. Specifically, for each training exam- ple (x, y), the IB loss is upper bounded by: LV IB= Ex∼p(x) Ez∼p(z|x)[− log q(y|z)] +β · KL(p(z|x), p0(z)) (2) where, q(y|z) is a variational approximation to p(y|z). p0(z), a specified prior distribution for latent re...

  21. [30]

    Qwen-audio: Advancing universal audio under- standing via unified large-scale audio-language models,

    Y . Chu et al., “Qwen-audio: Advancing universal audio under- standing via unified large-scale audio-language models,” arXiv preprint, 2023

    2023

  22. [31]

    Enhancing multimodal emotion recogni- tion through asr error compensation and llm fine-tuning,

    J. Kyung et al., “Enhancing multimodal emotion recogni- tion through asr error compensation and llm fine-tuning,” in INTERSPEECH, 2024

    2024

  23. [32]

    Large language model based generative er- ror correction: A challenge and baselines for speech recognition, speaker tagging, and emotion recognition,

    C.-H. H. Yang et al., “Large language model based generative er- ror correction: A challenge and baselines for speech recognition, speaker tagging, and emotion recognition,” in SLT, 2024

    2024

  24. [33]

    Wavcaps: A chatgpt-assisted weakly-labelled au- dio captioning dataset for audio-language multimodal research,

    X. Mei et al., “Wavcaps: A chatgpt-assisted weakly-labelled au- dio captioning dataset for audio-language multimodal research,” TASLP, 2024

    2024

  25. [34]

    Training audio captioning models without audio,

    S. Deshmukh et al., “Training audio captioning models without audio,” in ICASSP, 2024

    2024

  26. [35]

    Aligncap: Aligning speech emotion captioning to human preferences,

    Z. Liang et al., “Aligncap: Aligning speech emotion captioning to human preferences,” arXiv preprint, 2024

    2024

  27. [36]

    Clap4emo: Chatgpt-assisted speech emotion retrieval with natural language supervision,

    W.-C. Lin et al., “Clap4emo: Chatgpt-assisted speech emotion retrieval with natural language supervision,” in ICASSP, 2024

    2024

  28. [37]

    Lora: Low-rank adaptation of large language mod- els,

    E. J. Hu et al., “Lora: Low-rank adaptation of large language mod- els,” arXiv preprint, 2021

    2021

  29. [38]

    The information bottleneck method,

    N. Tishby et al., “The information bottleneck method,” Allerton, 2000

    2000

  30. [39]

    Deep variational information bottleneck,

    A. Alemi et al., “Deep variational information bottleneck,”ICLR, 2017

    2017

  31. [40]

    Auto-encoding variational bayes,

    D. P. Kingma et al., “Auto-encoding variational bayes,” ICLR, 2014

    2014

  32. [41]

    Bayesian learning for deep neural network adapta- tion,

    X. Xie et al., “Bayesian learning for deep neural network adapta- tion,” TASLP, 2021

    2021

  33. [42]

    Bayesian learning of lf-mmi trained time delay neu- ral networks for speech recognition,

    S. Hu et al., “Bayesian learning of lf-mmi trained time delay neu- ral networks for speech recognition,” TASLP, 2021

    2021

  34. [43]

    Hubert: Self-supervised speech representation learning by masked prediction of hidden units,

    W. Hsu et al., “Hubert: Self-supervised speech representation learning by masked prediction of hidden units,” TASLP, 2021

    2021

  35. [45]

    Llama-omni: Seamless speech interaction with large language models,

    Q. Fang et al., “Llama-omni: Seamless speech interaction with large language models,” arXiv preprint, 2024

    2024

  36. [46]

    The llama 3 herd of models,

    A. Dubey et al., “The llama 3 herd of models,” arXiv preprint, 2024

    2024

  37. [47]

    Speechcraft: A fine-grained expressive speech dataset with natural language description,

    Z. Jin et al., “Speechcraft: A fine-grained expressive speech dataset with natural language description,” in ACM MM, 2024

    2024

  38. [48]

    Gigaspeech: An evolving, multi-domain asr cor- pus with 10,000 hours of transcribed audio,

    G. Chen et al., “Gigaspeech: An evolving, multi-domain asr cor- pus with 10,000 hours of transcribed audio,”arXiv preprint, 2021

    2021

  39. [49]

    Emosec: Emotion recognition from scene context,

    S. Thuseethan et al., “Emosec: Emotion recognition from scene context,” NEUROCOMPUTING, 2022

    2022

  40. [50]

    IEMOCAP: interactive emotional dyadic motion capture database,

    C. Busso et al., “IEMOCAP: interactive emotional dyadic motion capture database,” LANG RESOUR EV AL, 2008

    2008

  41. [51]

    Meld: A multimodal multi-party dataset for emo- tion recognition in conversations,

    S. Poria et al., “Meld: A multimodal multi-party dataset for emo- tion recognition in conversations,”arXiv preprint, 2018

    2018

  42. [52]

    Emobox: Multilingual multi-corpus speech emotion recognition toolkit and benchmark,

    Z. Ma et al., “Emobox: Multilingual multi-corpus speech emotion recognition toolkit and benchmark,” INTERSPEECH, 2024

    2024

  43. [53]

    Disentanglement network: Disentangle the emo- tional features from acoustic features for speech emotion recogni- tion,

    Z. Yuan et al., “Disentanglement network: Disentangle the emo- tional features from acoustic features for speech emotion recogni- tion,” in ICASSP, 2024

    2024

  44. [54]

    How first-and second-language emotion words influence emotion perception in swedish–english bilinguals,

    M.-F. Champoux-Larsson et al., “How first-and second-language emotion words influence emotion perception in swedish–english bilinguals,” BILING-LANG COGN, 2022

    2022

  45. [55]

    Blsp-emo: Towards empathetic large speech- language models,

    C. Wang et al., “Blsp-emo: Towards empathetic large speech- language models,” EMNLP, 2024

    2024

  46. [56]

    Speech emotion recognition using decomposed speech via multi-task learning,

    J.-H. Hsu et al., “Speech emotion recognition using decomposed speech via multi-task learning,” in INTERSPEECH, 2023

    2023

  47. [57]

    Wavlm: Large-scale self-supervised pre-training for full stack speech processing,

    S. Chen et al., “Wavlm: Large-scale self-supervised pre-training for full stack speech processing,” IEEE J. Sel. Top. Signal Process., 2022

    2022

  48. [58]

    wav2vec 2.0: A framework for self-supervised learning of speech representations,

    A. Baevski et al., “wav2vec 2.0: A framework for self-supervised learning of speech representations,” Adv. Neural Inf. Process. Syst., 2020

    2020

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.