Pith. sign in

REVIEW 4 major objections 5 minor 37 references

Incorporating Contextual Paralinguistic Understanding in Large Speech-Language Models

T0 review · 4 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Training a Speech-LLM on QA pairs that fuse transcripts with categorical and continuous emotion labels improves contextual-paralinguistic QA by 38.41% (LLM-judged) over a content-only baseline, and adding explicit emotion metadata pushes th

desk verdict Honest, useful recipe for speech-LLM empathetic reasoning, but the headline gain rests on a judge validated only for classification, not for the open-ended task it scores. read the letter →

arxiv 2508.07273 v1 pith:6XZC52XN submitted 2025-08-10 cs.CL cs.AIeess.AS

Qiongqiong Wang , Hardik B. Sailor , Jeremy H. M. Wong , Tianchi Liu , Shuo Sun , Wenyu Zhang , Muhammad Huzaifah , Nancy Chen
show 1 more author
Ai Ti Aw
This is my paper
classification cs.CLcs.AIeess.AS
keywords speech-languagemodelscontextualparalinguisticunderstandingemotionrecognitionempatheticreasoningquestion-answergenerationLLM-as-judgedimensionalannotationsspeech
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that large speech-language models—LLMs extended with speech input—can learn contextual, empathetic question answering from training data, not from architectural changes. The authors propose an implicit method that automatically generates 32,960 QA pairs from transcripts plus categorical emotion labels (happy, sad, angry) and continuous dimensions (valence, arousal, dominance), and an explicit method that appends time-stamped emotion estimates to the prompt. On a human-annotated contextual-paralinguistic QA benchmark, the implicit training data raises the LLM-judged score from 41.00 to 56.75, a 38.41% relative gain, and combining it with explicit emotion metadata reaches 59.87, about a 46% relative gain. The paper also proposes estimated accuracy and F1 metrics to validate the LLM judge on deterministic emotion questions. A sympathetic reader would take the contribution as a data-centric recipe for making speech assistants emotionally aware without expensive manual annotation.

What carries the argument

The load-bearing object is the enhanced CPQA generation pipeline: emotion-rich speech clips are condensed and filtered, word-level transcripts are aligned with categorical emotion labels and continuous valence-arousal-dominance estimates, and an LLM prompt generates diverse QA pairs that must combine content and paralinguistic context, including open-ended questions like 'Why is the speaker angry?'. The paper's added ingredients are feeding the dimensional emotion scores into the generation prompt and scaling the generation to a training-sized corpus. The second mechanism is explicit emotion-metadata injection, where time-stamped emotion labels are appended to the question prompt so the LLM

What would settle it

Have independent human annotators, blind to system identity, rate the open-ended CPQA responses from the baseline and the proposed system, or use an LLM judge that was never used to generate training data. If the human ratings or the independent judge do not reproduce the roughly 38% relative gain, the reported improvement is a judge artifact rather than evidence of contextual paralinguistic understanding.

Watch

Extended reading notes

Core claim

The paper's central claim is that contextual paralinguistic reasoning in Speech-LLMs is learnable from generated QA data: training on pairs that force the model to reason jointly over what was said and how it was said is enough to produce large gains. The key comparison is the baseline system trained only on ASR, gender, and emotion QA versus the system additionally trained on the proposed PQA* and CPQA data, which scores 56.75 versus 41.00 on the human-annotated set, with gains on all question types. The paper further shows that explicitly providing time-stamped emotion metadata at inference compensates for a model that never learned to extract affect from audio, and using that metadata at

Load-bearing premise

The reported improvement rests on assuming that a GPT-4o judge gives valid, unbiased scores for open-ended empathetic responses, even though the training QA pairs were generated by the same model and the judge was only checked against deterministic emotion labels.

Editorial extensions

If this is right

  • Adding the generated PQA* and CPQA sets to the baseline training mix raises the human-annotated CPQA score from 41.00 to 56.75, a 38.41% relative improvement.
  • Explicit emotion metadata at inference alone improves baseline contextual+emotion question performance by 35.84%, indicating that external affective cues can substitute for missing paralinguistic understanding.
  • Using emotion metadata in both training and inference achieves the best emotion-question performance and an overall human-set score of 59.87 (a 46.03% relative gain over baseline), though it slightly lowers contextual-only and contextual+gender scores.
  • The implicit training data also improves contextual-only and contextual+gender questions, so the gain reflects broader contextual reasoning, not only emotion labeling.
  • LLM-judge scores on deterministic emotion questions correlate with estimated weighted accuracy and F1, so classification-style checks can cross-validate the judge on the paralinguistic subset.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the same model generates the training QA pairs and scores the evaluation responses, part of the measured gain could be stylistic self-preference; a human-written evaluation set or a judge from a different model family would isolate this.
  • The explicit-metadata results suggest a deployment shortcut the paper does not claim: an external speech-emotion front-end could pass time-stamped estimates to a frozen LLM at inference and recover part of the empathetic reasoning without retraining.
  • A natural ablation the paper leaves implicit: regenerate CPQA with only categorical labels, dropping valence-arousal-dominance, and compare contextual+emotion scores; if they drop, the dimensional annotations are the active ingredient.
  • The gains on contextual-only questions raise the possibility that this data-generation recipe transfers to other audio-reasoning tasks, such as speaker intent or sarcasm, though the paper does not test that directly.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper addresses limitations of Speech-LLMs in contextual paralinguistic understanding. It proposes (1) an explicit method that injects time-stamped emotion metadata into prompts at training/inference, and (2) an implicit method that augments training with GPT-4o-generated QA pairs (PQA*, CPQA) built from speech transcripts plus categorical and dimensional emotion annotations. The central empirical claim is that the implicit method improves LLM-judged performance on a human-annotated CPQA evaluation set by 38.41% (S30 vs. S10, Table III), and by 46.02% when combined with explicit metadata at training and inference. The paper also proposes estimated accuracy and F1 metrics to interpret LLM judge scores on deterministic classification questions, and reports question-type-wise analyses (C, CE, CG) and results on emotion-PQA benchmarks.

Significance. If the reported improvements are genuine, the data-generation pipeline would be practically useful: it scales QA data for Speech-LLMs while combining contextual, categorical, and dimensional paralinguistic cues, and the explicit metadata injection offers a way to probe the upper bound of emotion-grounding. The paper also ships a released evaluation set, uses a reproducible framework (MERaLiON-AudioLLM, AudioBench), and attempts to validate LLM judges beyond raw scores. However, the central 38.41% improvement rests on a GPT-4o judge for open-ended CPQA responses that is not validated for that setting, and the training data were generated by the same model. The paper's own admission that the LLM-generated evaluation set is compromised (§V-B3) further weakens the evaluation story. The core idea is promising, but the main empirical claim currently lacks a validity check on its key metric, so the manuscript needs substantial revision.

major comments (4)
  1. [§V-A3, §V-B4, Table III] The central claim (S30 improves over S10 by 38.41% on the human-annotated CPQA set) is measured only by gpt4o-as-judge for open-ended responses. The validation in §V-B4 is limited to deterministic emotion/gender classification questions, where outputs are converted to labels by Algorithm 1 and compared to ground truth. That does not establish that GPT-4o reliably scores contextual open-ended empathetic reasoning, which is exactly what Table III measures. The risk is compounded because all proposed training QA (PQA*, CPQA) is generated by GPT-4o, so S20/S30 responses may be closer in style and phrasing to GPT-4o's own outputs, while S10 was trained on template-based QA. A judge that prefers such outputs would inflate the reported gain independently of any real paralinguistic understanding. The human annotators wrote the reference QA pairs but did not score model outputs. I would need eith
  2. [§V-A2, §II-D] The CPQA training data and the human-annotated evaluation set are both produced by the same data-condensation pipeline from the same in-house movie/TV corpus: 4,740 clips for training and 480 clips for evaluation. The paper does not report whether any clips, utterances, or QA pairs overlap between the training and evaluation sets, nor whether the same generation pipeline and similar prompts are used. If the evaluation clips come from the same distribution as training and the QA pairs resemble the GPT-4o-generated training questions, then S30's advantage may partly reflect memorization or distribution overlap rather than generalization. Please state explicitly the disjointness of speakers/clips/QA pairs and, if possible, evaluate on an external corpus.
  3. [Table III, §V-A1] No statistical significance, confidence intervals, or multiple-seed results are reported. The systems differ in training data composition, but there is only one run per configuration; training steps are fixed at 120,000, but no seeds are given. Given that the main improvements are 52.06 vs. 41.00 and 56.75 vs. 41.00, the reader cannot assess whether these differences are stable. Please report variance across at least 3 seeds or bootstrap CIs, and ideally a paired significance test on the evaluation items.
  4. [§V-B3, Table III] The paper states that the LLM-generated evaluation set is compromised because direct emotion questions can make the answer identifiable from injected time-stamped emotion metadata. However, results on this set are still reported in Table III and used in Fig. 2 to support conclusions. The compromised set should either be excluded from the main results or replaced with a version that passes stricter generation controls. This is not just a caveat: it affects the credibility of any comparison that includes the 'LLM' column.
minor comments (5)
  1. [§II-D] The bullet 'Training-Scale CPQA Data using better prompt' is grammatically awkward; consider 'Training-scale CPQA data with an improved prompt'.
  2. [§V-A2] 'The detailed statistics of QA pairs generated from each dataset as well as the task wise are shown in Table I' is unclear. Consider rewriting to separate per-corpus and per-task statistics.
  3. [Fig. 2] The caption does not define what 'weighted average score' refers to, nor does it label the y-axis and the systems clearly. Please make the figure self-contained.
  4. [§V-B4] The sentence 'Note that we used all emotion categories 8 for IEMOCAP and 9 for MSP-Podcast' is ambiguous. Spell out the category sets and how they map to the models' label spaces.
  5. [References] Ref. [23] is the authors' prior Interspeech paper that introduces CPQA; please ensure the relationship and differences are clear beyond the short description in §II-C, especially which parts are new in this submission.

Circularity Check

1 steps flagged · score 4.0 of 10

Headline CPQA gain rests on a GPT-4o judge that also generated the training data; the judge is validated only on classification-style questions, leaving the main open-ended metric in a self-referential loop.

  1. fitted input called prediction [Section II-D / V-A2 (training data) and Section V-A3 (evaluation), Table III]
    "The proposed PQA* and CPQA datasets are generated using the GPT-4o API (Azure version 2024-07-01-preview) ... we employ AudioBench [18] for the assessment that uses gpt4o-as-judge to evaluate task performance."

    The same model (GPT-4o) both generates the proposed training QA pairs and serves as the judge for the central CPQA metric. The Speech-LLM is trained to imitate GPT-4o-generated answers, so its outputs become more GPT-4o-like in style and content; the GPT-4o judge then scores those outputs. The reported 38.41% gain (Table III) may therefore partly measure convergence to the judge's own output distribution rather than independent paralinguistic understanding. The paper validates the judge only on deterministic emotion/gender classification (Section IV, V-B4), not on the open-ended contextual responses that the headline score measures, so the training-to-judge loop is never broken.

full rationale

The paper's core contribution is an empirical training comparison: adding GPT-4o-generated PQA* and CPQA data to a baseline Speech-LLM and measuring CPQA performance. The human-annotated evaluation set provides some independent grounding, and the classification-metric validation (estimated accuracy/F1) is a genuine external check for deterministic emotion/gender questions. However, the headline 38.41% improvement is measured by a GPT-4o judge, and the proposed training data were also generated by GPT-4o. This creates a structural circularity in the evaluation: the model is fitted to imitate GPT-4o outputs, and then scored by GPT-4o. The paper's own validation of the judge is limited to classification tasks with estimated labels, not the open-ended contextual CPQA responses that the central claim depends on; the paper even concedes that its LLM-generated evaluation set is compromised by answer leakage. Self-citations to prior work [23] and own-benchmark [18] are used as infrastructure, but they are not the main circularity. No equation reduces the result to its inputs by construction, but the central metric is not independent of the training-data generator, so moderate circularity is present.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the quality of automatically generated training data, the validity of the LLM judge, and several hand-chosen filtering thresholds in the data condensation pipeline. These are not derived from first principles, and the paper does not provide independent evidence that the SER labels or the GPT-4o judge are reliable for the claimed open-ended task.

free parameters (3)
  • valence thresholds for SER consistency filter = [0,0.5), (0.5,1.0], [0.4,0.6] for negative, positive, neutral
    Hand-chosen ranges used to filter emotion labels during data condensation; they determine which clips enter the CPQA training set.
  • emotion occurrence minimum counts = [3,3,2,2,1,1] for angry, happy, sad, surprised, disgusted, fearful
    Hand-chosen thresholds to balance and enlarge the emotion-rich training data; they affect the composition of the CPQA set.
  • training steps and learning rate = 120,000 steps, lr=1e-4
    Fixed hyperparameters that ensure fair comparison between systems; the choice of 120,000 steps is not justified by convergence analysis.
assumptions (4)
  • domain assumption Emotion labels and dimensions estimated by SER tools (Emotion2Vec and a dimensional model) are sufficiently accurate to serve as training signals for empathetic reasoning.
    Invoked throughout Section II-C/D and V-A2; the entire data-generation pipeline, and hence the implicit method, depends on this.
  • domain assumption GPT-4o-generated QA pairs are high-quality training data that teach contextual-paralinguistic reasoning.
    The implicit method relies on the quality of automatically generated QA pairs; no human validation of the training QA quality is reported.
  • domain assumption A GPT-4o judge produces a valid measure of open-ended empathetic reasoning in CPQA.
    The main results are all LLM-judged. The paper's own validation (Section IV) covers only deterministic classification questions, not open-ended contextual reasoning.
  • standard math Cosine similarity with paraphrase-MiniLM-L6-v2 is a valid semantic matching for converting free-form answers into classification labels.
    Used in Algorithm 1 to compute accuracy and F1; this is an accepted technique but an unproved modeling choice for the specific task.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Incorporating Contextual Paralinguistic Understanding in Large Speech-Language Models." pith.science (2026). https://pith.science/paper/6XZC52XN

@misc{pith2026250807273,
  author       = {Pith},
  title        = {Pith review of: Incorporating Contextual Paralinguistic Understanding in Large Speech-Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6XZC52XN}},
  note         = {Machine review of arXiv:2508.07273}
}
read the original abstract

Current large speech language models (Speech-LLMs) often exhibit limitations in empathetic reasoning, primarily due to the absence of training datasets that integrate both contextual content and paralinguistic cues. In this work, we propose two approaches to incorporate contextual paralinguistic information into model training: (1) an explicit method that provides paralinguistic metadata (e.g., emotion annotations) directly to the LLM, and (2) an implicit method that automatically generates novel training question-answer (QA) pairs using both categorical and dimensional emotion annotations alongside speech transcriptions. Our implicit method boosts performance (LLM-judged) by 38.41% on a human-annotated QA benchmark, reaching 46.02% when combined with the explicit approach, showing effectiveness in contextual paralinguistic understanding. We also validate the LLM judge by demonstrating its correlation with classification metrics, providing support for its reliability.

Figures

Figures reproduced from arXiv: 2508.07273 by the authors.

Figure 1
Figure 1. Prompt for generating QA pairs from audio clips using both dimensional and categorical emotion annotations [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Impact of emotion metadata in training and inference prompt. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Distribution of original LLM scores (before scaling to 0–100) with [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

37 extracted references · 27 canonical work pages

  1. [1]

    Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models,

    Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou, “Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models,” arXiv preprint arXiv:2311.07919 , 2023

    arXiv 2023

  2. [2]

    Qwen2-audio technical report,

    Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, Chang Zhou, and Jingren Zhou, “Qwen2-audio technical report,” arXiv preprint arXiv:2407.10759, 2024

    arXiv 2024

  3. [3]

    GPT-4 technical report,

    Josh Achiama, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, and other, “GPT-4 technical report,” arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  4. [4]

    Salmonn: Towards generic hearing abilities for large language models,

    Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, and Chao Zhang, “Salmonn: Towards generic hearing abilities for large language models,” arXiv preprint arXiv:2310.13289, 2023

    arXiv 2023

  5. [5]

    Advancing large language models to capture varied speaking styles and respond properly in spoken conversations,

    Guan-Ting Lin, Cheng-Han Chiang, and Hung-Yi Lee, “Advancing large language models to capture varied speaking styles and respond properly in spoken conversations,” arXiv preprint arXiv:2402.12786 , 2024

    arXiv 2024

  6. [6]

    Paralinguistics- aware speech-empowered large language models for natural conversa- tion,

    Heeseung Kim, Soonshin Seo, Kyeongseok Jeong, Ohsung Kwon, Soyoon Kim, Jungwhan Kim, Jaehong Lee, Eunwoo Song, Myungwoo Oh, Jung-Woo Ha, Sungroh Yoon, and Kang Min Yoo, “Paralinguistics- aware speech-empowered large language models for natural conversa- tion,” in The Thirty-eighth Annual Conference on Neural Information Processing Systems (NeurIPS) , 2024

    work page 2024

  7. [7]

    Frozen large language models can perceive paralinguistic aspects of speech,

    Wonjune Kang, Junteng Jia, Chunyang Wu, Wei Zhou, Egor Lakomkin, Yashesh Gaur, Leda Sari, Suyoun Kim, Ke Li, Jay Mahadeokar, et al., “Frozen large language models can perceive paralinguistic aspects of speech,” arXiv preprint arXiv:2410.01162 , 2024

    arXiv 2024

  8. [8]

    BLSP-Emo: Towards empathetic large speech-language models,

    Chen Wang, Minpeng Liao, Zhongqiang Huang, Junhong Wu, Chengqing Zong, and Jiajun Zhang, “BLSP-Emo: Towards empathetic large speech-language models,” Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP) , 2024

    work page 2024

Show all 37 references
  1. [9]

    DeSTA: Enhancing speech language models through descriptive speech-text alignment,

    Ke-Han Lu, Zhehuai Chen, Szu-Wei Fu, He Huang, Boris Ginsburg, Yu- Chiang Wang, and Hung-yi Lee, “DeSTA: Enhancing speech language models through descriptive speech-text alignment,” in Interspeech, 2024, pp. 4159–4163

    2024

  2. [10]

    DeSTA2: Developing instruction-following speech language model without speech instruction-tuning data,

    Ke-Han Lu, Zhehuai Chen, Szu-Wei Fu, Chao-Han Huck Yang, Ja- gadeesh Balam, Boris Ginsburg, Yu-Chiang Wang, and Hung-yi Lee, “DeSTA2: Developing instruction-following speech language model without speech instruction-tuning data,” in IEEE International Con- ference on Acoustics...

    2025

  3. [11]

    Beyond silent letters: Amplifying LLMs in emotion recognition with vocal nuances,

    Zehui Wu, Ziwei Gong, Lin Ai, Pengyuan Shi, Kaan Donbekci, and Julia Hirschberg, “Beyond silent letters: Amplifying LLMs in emotion recognition with vocal nuances,” in Findings of the Association for Computational Linguistics: NAACL 2025 , 2025, pp. 2202–2218

    2025

  4. [12]

    CLAP4Emo: ChatGPT-Assisted Speech Emotion Retrieval with Natural Language Supervision,

    Wei-Cheng Lin, Shabnam Ghaffarzadegan, Luca Bondi, Abinaya Kumar, Samarjit Das, and Ho-Hsiang Wu, “CLAP4Emo: ChatGPT-Assisted Speech Emotion Retrieval with Natural Language Supervision,” in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2024...

    2024

  5. [13]

    SECap: Speech emotion captioning with large language model,

    Yaoxun Xu, Hangting Chen, Jianwei Yu, Qiaochu Huang, Zhiyong Wu, Shi-Xiong Zhang, Guangzhi Li, Yi Luo, and Rongzhi Gu, “SECap: Speech emotion captioning with large language model,” in Proceedings of the AAAI Conference on Artificial Intelligence , 2024, vol. 38, pp. 19323–19331

    2024

  6. [14]

    Empower typed descriptions by large language models for speech emotion recognition,

    Haibin Wu, Huang-Cheng Chou, Kai-Wei Chang, Lucas Goncalves, Jiawei Du, Jyh-Shing Roger Jang, Chi-Chun Lee, and Hung-Yi Lee, “Empower typed descriptions by large language models for speech emotion recognition,” in Asia Pacific Signal and Information Processing Association Annu...

    2024

  7. [15]

    V oxDialogue: Can spoken dialogue systems understand information be- yond words?,

    Xize Cheng, Ruofan Hu, Xiaoda Yang, Jingyu Lu, Dongjie Fu, Zehan Wang, Shengpeng Ji, Rongjie Huang, Boyang Zhang, Tao Jin, et al., “V oxDialogue: Can spoken dialogue systems understand information be- yond words?,” in International Conference on Learning Representations (ICLR), 2025

    2025

  8. [16]

    SIFT-50M: A large-scale multilingual dataset for speech instruction fine-tuning,

    Prabhat Pandey, Rupak Vignesh Swaminathan, KV Girish, Arunasish Sen, Jian Xie, Grant P Strimel, and Andreas Schwarz, “SIFT-50M: A large-scale multilingual dataset for speech instruction fine-tuning,” arXiv preprint arXiv:2504.09081, 2025

    2025 arXiv

  9. [17]

    Joint audio and speech understanding,

    Yuan Gong, Alexander H Liu, Hongyin Luo, Leonid Karlinsky, and James Glass, “Joint audio and speech understanding,” in IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). IEEE, 2023, pp. 1–8

    2023

  10. [18]

    Audiobench: A universal benchmark for audio large language models,

    Bin Wang, Xunlong Zou, Geyu Lin, Shuo Sun, Zhuohan Liu, Wenyu Zhang, Zhengyuan Liu, AiTi Aw, and Nancy F Chen, “Audiobench: A universal benchmark for audio large language models,” Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for ...

    2025

  11. [19]

    Dynamic-superb: Towards a dy- namic, collaborative, and comprehensive instruction-tuning benchmark for speech,

    Chien-Yu Huang, Ke-Han Lu, Shih-Heng Wang, Chi-Yuan Hsiao, Chun- Yi Kuan, Haibin Wu, Siddhant Arora, Kai-Wei Chang, Jiatong Shi, Yifan Peng, Roshan Sharma, Shinji Watanabe, Bhiksha Ramakrishnan, Shady Shehata, and Hung-Yi Lee, “Dynamic-superb: Towards a dy- namic, collaborativ...

  12. [20]

    AIR-Bench: Benchmarking large audio-language models via generative comprehension,

    Qian Yang, Jin Xu, Wenrui Liu, Yunfei Chu, Ziyue Jiang, Xiaohuan Zhou, Yichong Leng, Yuanjun Lv, Zhou Zhao, Chang Zhou, et al., “AIR-Bench: Benchmarking large audio-language models via generative comprehension,” Proceedings of the 62nd Annual Meeting of the Association for Com...

    1979

  13. [21]

    Listen, think, and understand,

    Yuan Gong, Hongyin Luo, Alexander H Liu, Leonid Karlinsky, and James Glass, “Listen, think, and understand,” in International Confer- ence on Learning Representations (ICLR) , 2024

    2024

  14. [22]

    MMAU: A massive multi-task audio understanding and reasoning benchmark,

    S Sakshi, Utkarsh Tyagi, Sonal Kumar, Ashish Seth, Ramaneswaran Selvakumar, Oriol Nieto, Ramani Duraiswami, Sreyan Ghosh, and Dinesh Manocha, “MMAU: A massive multi-task audio understanding and reasoning benchmark,” in International Conference on Learning Representations (ICLR), 2025

    2025

  15. [23]

    Contextual paralinguistic data creation for multi-modal Speech-LLM: Data condensation and spoken QA generation,

    Qiongqiong Wang, Hardik B Sailor, Tianchi Liu, and Ai Ti Aw, “Contextual paralinguistic data creation for multi-modal Speech-LLM: Data condensation and spoken QA generation,” in Proc. Interspeech, 2025

    2025

  16. [24]

    WavLM: Large-scale self-supervised pre-training for full stack speech processing,

    Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, et al., “WavLM: Large-scale self-supervised pre-training for full stack speech processing,” IEEE Journal of Selected Topics in Signal Process- ing, v...

    2022

  17. [25]

    emotion2vec: Self-supervised pre-training for speech emotion representation,

    Ziyang Ma, Zhisheng Zheng, Jiaxin Ye, Jinchao Li, Zhifu Gao, Shiliang Zhang, and Xie Chen, “emotion2vec: Self-supervised pre-training for speech emotion representation,” Findings of the Association for Computational Linguistics (ACL) , 2024

    2024

  18. [26]

    Dawn of the transformer era in speech emotion recognition: Closing the valence gap,

    Johannes Wagner, Andreas Triantafyllopoulos, Hagen Wierstorf, Maxi- milian Schmitt, Felix Burkhardt, Florian Eyben, and Bj ¨orn W Schuller, “Dawn of the transformer era in speech emotion recognition: Closing the valence gap,” IEEE Transactions on Pattern Analysis and Machine I...

    2023

  19. [27]

    Whis- perX: Time-accurate speech transcription of long-form audio,

    Max Bain, Jaesung Huh, Tengda Han, and Andrew Zisserman, “Whis- perX: Time-accurate speech transcription of long-form audio,” in Proc. Interspeech, 2023

    2023

  20. [28]

    Sentence-bert: Sentence embeddings using siamese bert-networks,

    Nils Reimers and Iryna Gurevych, “Sentence-bert: Sentence embeddings using siamese bert-networks,” in Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP) . 11 2019, Association for Computational Linguistics

    2019

  21. [29]

    MERaLiON-AudioLLM: Technical report,

    Yingxu He, Zhuohan Liu, Shuo Sun, Bin Wang, Wenyu Zhang, Xunlong Zou, Nancy F Chen, and Ai Ti Aw, “MERaLiON-AudioLLM: Technical report,” arXiv preprint arXiv:2412.09818 , 2024

    2024 arXiv

  22. [30]

    Robust speech recognition via large- scale weak supervision,

    Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever, “Robust speech recognition via large- scale weak supervision,” Proceedings of the 40th International Confer- ence on Machine Learning (ICML) , 2023

    2023

  23. [31]

    Gemma: Open models based on gemini research and technol- ogy,

    Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivi `ere, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, and Hussenot L ´eonard others, “Gemma: Open models based on gemini research and technol- ogy,” arXiv p...

    2024 arXiv

  24. [32]

    Advancing Singlish understanding: Bridging the gap with datasets and multimodal models,

    Bin Wang, Xunlong Zou, Shuo Sun, Wenyu Zhang, Yingxu He, Zhuohan Liu, Chengwei Wei, Nancy F Chen, and AiTi Aw, “Advancing Singlish understanding: Bridging the gap with datasets and multimodal models,” arXiv preprint arXiv:2501.01034 , 2025

    2025 arXiv

  25. [33]

    Building the Singapore English national speech corpus,

    Jia Xin Koh, Aqilah Mislan, Kevin Khoo, Brian Ang, Wilson Ang, Charmaine Ng, and Ying-Ying Tan, “Building the Singapore English national speech corpus,” in Proc. Interspeech, 2019, pp. 321–325

    2019

  26. [34]

    Librispeech: An ASR corpus based on public domain audio books,

    Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur, “Librispeech: An ASR corpus based on public domain audio books,” in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2015, pp. 5206–5210

    2015

  27. [35]

    IEMOCAP: interactive emotional dyadic motion capture database,

    Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Ebrahim (Abe) Kazemzadeh, Emily Mower Provost, Samuel Kim, Jeannette N. Chang, Sungbok Lee, and Shrikanth S. Narayanan, “IEMOCAP: interactive emotional dyadic motion capture database,” Language Resources and Evaluation, vol. 42, pp. 3...

    2008

  28. [36]

    V oxceleb: A large-scale speaker identification dataset,

    Arsha Nagrani, Joon Son Chung, and Andrew Zisserman, “V oxceleb: A large-scale speaker identification dataset,” in Proc. Interspeech, 2017, pp. 2616–2620

    2017

  29. [37]

    The MSP-Podcast corpus for speech emotion recognition,

    Carlos Busso, Siddharth Narayanan, Emily Mower Provost, Yue Zhang, Asterios Matsoukas, and Najim Dehak, “The MSP-Podcast corpus for speech emotion recognition,” IEEE Transactions on Affective Computing, 2023

    2023

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.