Pith. sign in

REVIEW 4 major objections 5 minor 40 references

Contextual Paralinguistic Data Creation for Multi-Modal Speech-LLM: Data Condensation and Spoken QA Generation

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that an automated pipeline can generate contextual paralinguistic QA pairs from raw speech that evaluate speech-LLMs about as well as human-written QA.

desk verdict Useful pipeline and a released benchmark, but the 'strong correlation' claim is not demonstrated by the reported means. read the letter →

arxiv 2505.13338 v2 pith:TIZJQZRG submitted 2025-05-19 cs.CL cs.AIeess.AS

Qiongqiong Wang , Hardik B. Sailor , Tianchi Liu , Ai Ti Aw This is my paper · ORCID
classification cs.CLcs.AIeess.AS
keywords speech-LLMparalinguisticcontextualreasoningemotionrecognitiondatacondensationspokenQAgenerationempatheticevaluationdataset
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that a fully automated pipeline can turn unlabeled, in-the-wild speech into a benchmark of contextual paralinguistic QA pairs (questions and answers that tie what was said to how it was said, e.g., the speaker's emotion) that is as useful for evaluating speech-LLMs—large language models that take audio as input—as a human-written benchmark. Its condensation stage uses a hybrid of categorical and dimensional speech emotion recognition to filter audio down to reliably labeled, emotionally balanced samples. Its generation stage prompts a text LLM with word-level transcripts plus time-aligned emotion and gender labels to write the QA pairs. The central evidence is that an open speech-LLM receives comparable judge scores on the machine-written and human-written QA sets, so the authors conclude LLM-generated QA is a viable evaluation tool. If this is right, it removes the main bottleneck—expensive human annotation—for building empathetic speech-LLM training and evaluation data.

What carries the argument

The load-bearing mechanism is a two-stage pipeline. First, data condensation combines a categorical emotion recognizer with a dimensional valence estimator: sub-segments pass only if the categorical emotion's sentiment class agrees with the valence value, and whole segments pass only if enough sub-segments share one emotion label, giving a balanced, high-confidence pool. Second, the surviving audio is transcribed with word-level timestamps and each word is aligned to its overlapping emotion and gender labels; a text LLM is then prompted with the transcript and word-level paralinguistic metadata to generate CPQA pairs. The evaluation loop uses a separate judge LLM with two scoring prompts, one from an existing benchmark and a refined version that incorporates paralinguistic information, and takes the better of the first and last thirty seconds of each clip. These components together are what let the framework replace human annotation at both the labeling and question-writing stages.

What would settle it

Have human annotators label the emotion of each of the 480 released speech samples and compare those labels with the pseudo labels that drove QA generation; if many QA pairs reference an emotion the human labelers do not hear, the claim that the pipeline grounds QA in real paralinguistic content is weakened. A second check is to evaluate several speech-LLMs on both the machine-written and human-written QA sets and see whether their rankings agree; if rankings diverge, LLM-generated QA is not yet a substitute for human QA.

Watch

Extended reading notes

Core claim

The paper's central claim is that the proposed data creation framework produces contextual paralinguistic QA (CPQA) datasets whose quality is close enough to human-authored CPQA that LLM-generated QA can serve as a viable tool for evaluating speech-LLMs. The authors support this by generating 2,647 QA pairs with an LLM from 480 condensed speech samples, having two human annotators write a comparison set on the same audio, and scoring the answers of a speech-LLM with two different judge LLMs under two prompts. Across judge and prompt combinations, the machine-written set scores within about one to three points of the human set (for example, 53.86 versus 52.29 under one judge and 60.28 versus 59.46 under another). The paper also reports that the evaluation reveals the speech-LLM's weakness on empathetic reasoning, which is put forward as evidence that such datasets are needed.

Load-bearing premise

The load-bearing premise is that the automatically estimated emotion labels, reported at about 51% accuracy and 29% unweighted accuracy on the tuning set, remain accurate enough after condensation for the generated questions and answers to describe emotions the audio actually contains.

Editorial extensions

If this is right

  • Large-scale CPQA training sets for empathetic speech-LLMs can be produced from raw in-the-wild audio without human emotion annotation or manual question writing.
  • Evaluation of speech-LLMs can be automated end-to-end, allowing rapid comparison of models on contextual paralinguistic reasoning.
  • The released 480-sample benchmark gives the community a fixed reference set for measuring progress on emotion- and context-aware spoken QA.
  • Benchmark results built this way will keep surfacing the specific failure mode of empathetic reasoning in current speech-LLMs, guiding training-data priorities.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the QA generator is a text LLM, the framework's ceiling may be set by how faithfully the transcript plus aligned labels represent the audio; a testable extension could feed richer acoustic descriptors, such as arousal or pitch, into the prompt.
  • The reported pseudo-label accuracy implies that some fraction of QA pairs rests on emotion labels a human would dispute; a natural check is to measure how QA quality changes when the condensation thresholds are relaxed or tightened.
  • The same condensation-plus-generation pattern could be applied to other paralinguistic attributes, such as speaking style, speaker turns, or deception cues, once reliable detectors for those attributes exist.
  • Taking the higher score over first and last 30-second windows may inflate absolute scores on long clips; comparing model rankings rather than absolute scores would be a more robust use of the benchmark.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper presents a framework for automatically generating contextual paralinguistic question-answer (CPQA) pairs from in-the-wild speech. The pipeline first condenses speech data using an ensemble of categorical and dimensional emotion recognition models, filtering for SER consistency and emotion occurrence. It then uses WhisperX for word-level alignment and GPT-4o to generate QA pairs from transcripts enriched with pseudo emotion and gender labels. The authors release an evaluation set of 480 audio samples with 2,647 generated QA pairs, compare its statistics to a human-annotated QA set, and evaluate the speech-LLM Qwen2-Audio-7B-Instruct using both sets under two LLM judges (Llama-70B and ChatGPT). The central claim is that LLM-generated QA shows a 'strong correlation' with human-generated QA, and thus can serve as a viable evaluation tool for speech-LLMs.

Significance. If the central validation claim were properly established, the framework would be a valuable contribution: it addresses a real data scarcity for empathetic speech-LLM training and evaluation, and the open-source dataset is a useful community resource. The idea of combining categorical and dimensional SER for data condensation is sensible, and the alignment of word-level transcripts with paralinguistic metadata is a practical enabler for LLM-based QA generation. The paper also does a service by revealing weaknesses of current speech-LLMs on empathetic reasoning. However, the evidence presented for the load-bearing claim of correlation is currently insufficient, which limits the significance of the work until the validation is strengthened.

major comments (4)
  1. [Section 4, Table 2; Abstract] The paper claims a 'strong correlation' between LLM-generated and human-generated QA sets, but no correlation statistic is reported. Table 2 gives only mean judge scores (e.g., 53.86 vs. 52.29 and 60.28 vs. 59.46). Equal or similar means do not imply correlation: two sets can have identical average scores while ranking individual QA pairs very differently, which would make the LLM-generated set unreliable for evaluating a speech-LLM on specific reasoning failures. The authors must report per-sample agreement, a rank correlation (e.g., Spearman) or Pearson correlation with confidence intervals, and a significance test. Without such statistics, the central claim in the abstract and Section 4 is unsubstantiated.
  2. [Section 3.1] The pseudo emotion labels used to ground the generated QA are acknowledged to be noisy: the SER ensemble achieves 51.10% accuracy and 29.25% UWA on the tuning set, and the condensation filter improves UWA only to 33.65%. Since the QA generator is prompted with these labels, a substantial fraction of the generated QA pairs may describe emotions that are not actually present in the audio. The correlation claim could then be an artifact of both QA sets being easy or of judge leniency, rather than evidence of label fidelity. The authors should measure label accuracy on the condensed evaluation set itself (e.g., human verification of a sample of the 480 audio-label pairs) and report per-emotion agreement between the QA content and the audio content.
  3. [Section 3.2, Section 4, Figure 5] The generated QA pairs were not systematically human-rated. The paper mentions a manual review that found repetitive or irrelevant questions, but it does not report inter-annotator agreement, quality scores, or any structured human evaluation of the generated QA's relevance and answerability. At the same time, the evaluation uses LLM judges (Llama-70B and ChatGPT), and ChatGPT (the generator) is also one of the judges, which introduces shared model biases. The claim that LLM-generated QA is a viable evaluation tool should be supported by a human evaluation of QA quality and by at least one judge that is independent of the generator.
  4. [Section 4, Table 2] The reported mean scores are not accompanied by any measure of variability or significance. Differences such as 53.86 vs. 52.29 could be within noise, and no error bars, standard deviations, or statistical tests are provided. Moreover, the analysis does not break down performance by question type (e.g., emotion questions, contextual paralinguistic reasoning, content questions), even though Table 1 shows that the LLM-generated and human-generated sets differ substantially in their distributions of question types. Without a per-category comparison, it is impossible to tell whether the LLM-generated set is a faithful proxy on the dimensions that matter most, such as empathetic reasoning.
minor comments (5)
  1. [Section 3.2] The paper states that 80 samples were selected for each emotion category and that the total is 480 samples, implying six emotion categories. It earlier mentions six non-neutral categories (angry, disgusted, fearful, happy, sad, surprised). Please clarify whether the final evaluation set excludes a neutral category, and if so, why neutral speech was not included.
  2. [Section 2.1.3, Algorithm 1] The algorithm uses the symbols X_E and X_O for filter conditions, but the conditions are only fully defined in the following paragraph. The reader would benefit from a forward reference or a brief definition in the algorithm caption.
  3. [Figure 3] The heatmap in Figure 3 lacks explicit axis labels. It would be helpful to label the axes as 'x' and 'y' directly on the plot, and to note whether higher values correspond to better or worse UWA in the caption.
  4. [Section 4] The evaluation procedure takes both the first and last 30 seconds of each 30-to-60-second audio and uses the higher score. It should be stated explicitly whether this selection is done per QA pair or per question, and how often the two windows disagree.
  5. [References] There is a typographical spacing issue in reference [31]: 'V oxCeleb2' should be 'VoxCeleb2'.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the core validation is anchored by external human-generated QA and independent judges; weaknesses are statistical, not definitional.

full rationale

The paper's derivation chain is not circular. The central validation claim is that LLM-generated CPQA can serve as a viable evaluation tool for speech-LLMs, supported by comparing judge scores on LLM-generated QA against human-generated QA for the same 480 speech samples. The human-generated set is an independent anchor: it is not produced by the framework, and no framework parameter is fitted to the comparison outcome. The pseudo-label accuracy of 51.10% and 29.25% UWA reported in Section 3.1 is an empirical data-quality limitation, not a case where the predicted result is defined in terms of the input labels. Similarly, the absence of a correlation statistic in Table 2 and the claim of 'strong correlation' in the abstract are evidentiary weaknesses, not definitional circularity, because the paper does not construct the conclusion from its own fitted quantities. The use of ChatGPT as both QA generator and judge is a methodological confound, but the judge scores both QA sets under the same prompts, so the LLM-versus-human comparison is not forced by construction. Self-citations such as [5], [6], and [14] appear in background and prompt selection and are not load-bearing for the main result; no authors-only theorem is invoked to forbid alternatives. No equation in the paper reduces an output to an input, and no fitted parameter is renamed as a prediction. Therefore no circular step can be identified under the required standard of quoting a specific reduction.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central claim depends on transferred SER models, hand-tuned condensation thresholds, and implicit assumptions about LLM judges and QA fidelity. Most parameters are validated on a different corpus (SG TV/Movie) than the released benchmark, and no statistical confidence is provided for the evaluation comparison.

free parameters (5)
  • valence threshold x = 0.5
    v_pos,min=x and v_neg,max=1-x; optimized on SG TV/Movie dataset, achieving UWA 33.65% (Section 3.1).
  • valence threshold y = 0.4
    v_neu,min=y and v_neu,max=1-y; optimized on SG TV/Movie dataset (Section 3.1).
  • occurrence threshold alpha = [10,10,4,4,2,3]
    Per-emotion minimum count of consistent sub-segment labels for angry, disgusted, fearful, happy, sad, surprised; chosen by hand (Section 3.1).
  • minimum audio length tau = 30 seconds
    Pre-determined filter to ensure sufficient reasoning context (Section 2.1.3).
  • SER window size t and delta t = t=2s, delta t=1s (emotion); t=2s, delta t=0.5s (gender)
    Chosen for automatic labeling; not swept (Sections 2.1.1, 3.1).
assumptions (5)
  • domain assumption Emotion2vec ensemble reliably estimates discrete emotion categories on Singapore English in-the-wild speech
    The ensemble is tuned on the internal SG TV/Movie dataset (accuracy 51.10%, UWA 29.25%) and assumed to transfer to the YouTube benchmark (Section 3.1).
  • domain assumption Valence model fine-tuned on MSP-Podcast transfers to target speech
    The wav2vec2-based valence model trained on MSP-Podcast v1.7 is applied without domain adaptation (Section 2.1.2).
  • domain assumption Consistency filtering between categorical and dimensional SER removes unreliable labels while retaining true emotion distribution
    The filter in Algorithm 1 combines category and valence; no oracle study validates that low-consistency samples are actually mislabeled.
  • domain assumption LLM-generated QA pairs from transcripts and word-level metadata are a faithful proxy for human-authored contextual paralinguistic QA
    Validated only indirectly through comparable judge scores on one speech-LLM, not through direct QA quality assessment (Section 4).
  • domain assumption LLM judges (Llama-70B and ChatGPT-4o) give unbiased capability scores for speech-LLM answers
    Both judge prompts are text-LLMs; no calibration against ground-truth human ratings is reported (Section 4).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Contextual Paralinguistic Data Creation for Multi-Modal Speech-LLM: Data Condensation and Spoken QA Generation." pith.science (2026). https://pith.science/paper/TIZJQZRG

@misc{pith2026250513338,
  author       = {Pith},
  title        = {Pith review of: Contextual Paralinguistic Data Creation for Multi-Modal Speech-LLM: Data Condensation and Spoken QA Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TIZJQZRG}},
  note         = {Machine review of arXiv:2505.13338}
}
read the original abstract

Current speech-LLMs exhibit limited capability in contextual reasoning alongside paralinguistic understanding, primarily due to the lack of Question-Answer (QA) datasets that cover both aspects. We propose a novel framework for dataset generation from in-the-wild speech data, that integrates contextual reasoning with paralinguistic information. It consists of a pseudo paralinguistic label-based data condensation of in-the-wild speech and LLM-based Contextual Paralinguistic QA (CPQA) generation. The effectiveness is validated by a strong correlation in evaluations of the Qwen2-Audio-7B-Instruct model on a dataset created by our framework and human-generated CPQA dataset. The results also reveal the speech-LLM's limitations in handling empathetic reasoning tasks, highlighting the need for such datasets and more robust models. The proposed framework is first of its kind and has potential in training more robust speech-LLMs with paralinguistic reasoning capabilities.

Figures

Figures reproduced from arXiv: 2505.13338 by the authors.

Figure 1
Figure 1. Diagram of dataset creation framework. condensation and automated CPQA generation (see [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 3
Figure 3. Heatmap of UWA(%) performance on the SG TV/movie dataset for varying factors x and y. The triangle marks the best performance. filtering and occurrence filtering (see Algorithm 1), applied un￾der the following conditions: • SER consistency condition XE: Ensures consistency be￾tween sentiment class mapped from discrete emotion cate￾gories and valence values. A sub-segment si,j satisfies XE, a set of the following: ci… view at source ↗
Figure 4
Figure 4. Prompt for Generating QA Pairs from Audio Clips YouTube channels. 3.1. Preliminary study for framework parameters To set framework parameters, we conduct a preliminary study on an internal emotion dataset, SG TV/Movie dataset, compris￾ing 117k speech segments (120 hours) from Singaporean TV shows and movies, primarily in English with some Mandarin. The emotion labels are annotated by human annotators. In pre-process… view at source ↗
Figures from the paper (1 more)
Figure 5
Figure 5. Figure 5: The illustration of the evaluation pipeline [PITH_FULL_IMAGE:figures/full_fig_p004_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

40 extracted references · 24 canonical work pages

  1. [1]

    Recent speech-LLMs, such as GPT-4 [1], Qwen-audio [2, 3], SALMONN [4], and MERaLiON-AudioLLM [5, 6], have demonstrated remarkable performance in handling speech-based tasks

    Introduction Rapid advancements in large language models (LLMs) have sparked significant interest in multimodal models that inte- grate LLMs with speech modalities. Recent speech-LLMs, such as GPT-4 [1], Qwen-audio [2, 3], SALMONN [4], and MERaLiON-AudioLLM [5, 6], have demonstrated remarkable performance in handling speech-based tasks. Some speech- LLMs,...

    work page

  2. [2]

    Contextual Paralinguistic Data Creation for Multi-Modal Speech-LLM: Data Condensation and Spoken QA Generation

    Proposed data generation framework We propose a novel framework for CPQA dataset generation from in-the-wild speech data. The framework consists of data 1https://huggingface.co/datasets/MERaLiON/CPQA-Evaluation- Set arXiv:2505.13338v2 [cs.CL] 3 Jun 2025 Figure 1:Diagram of dataset creation framework. condensation and automated CPQA generation (see Figure ...

    work page Pith review arXiv 2025

  3. [3]

    What is the content in the audio from the text transcript?

    Evaluation dataset creation We construct a dataset by applying our proposed data cre- ation framework to speech data collected from top Singaporean 6https://learn.microsoft.com/en-us/azure/ai-services/openai/ Prompt for Generating QA Pairs from Audio Clips Generate diverse paralinguistic, content-based, and contextual reasoning QA pairs from a given audio...

    work page

  4. [4]

    Evaluation We validate the ChatGPT-generated CPQA set by evaluating Qwen2-Audio-7B-Instruct7 speech-LLM since it is the best per- forming open source model as shown in large scale MMAU evaluation [18]. To interpret the performance, we use Llama-3- 7https://huggingface.co/Qwen/Qwen2-Audio-7B-Instruct Judger (LLM 2) Proposed QA Generation Evaluation Perform...

    work page 2025

  5. [5]

    Additionally, the observed correlation between the two QA sets further supports the validity of LLM-generated CPQA as a reasonable approach to evaluate speech-LLMs

    work page

  6. [6]

    Our framework con- sists of pseudo paralinguistic label-based data condensation and LLM-based CPQA generation

    Summary We propose a novel framework for generating dataset with contextual paralinguistic QA (CPQA) pairs from in-the-wild speech data, addressing the scarcity of data available for developing empathetic speech-LLMs. Our framework con- sists of pseudo paralinguistic label-based data condensation and LLM-based CPQA generation. We release a benchmark datas...

    arXiv

  7. [7]

    Acknowledgement This research/project is supported by the National Research Foundation, Singapore, under its National Large Language Models Funding Initiative. Any opinions, findings, conclusions or recommendations expressed in this material are those of the author(s) and do not reflect the views of the National Research Foundation, Singapore

    work page

  8. [8]

    GPT-4 technical report,

    J. Achiama, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altmanet al., “GPT-4 technical report,”arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

Show all 40 references
  1. [9]

    Qwen-audio: Advancing universal audio understand- ing via unified large-scale audio-language models,

    Y . Chu, J. Xu, X. Zhou, Q. Yang, S. Zhang, Z. Yan, C. Zhou, and J. Zhou, “Qwen-audio: Advancing universal audio understand- ing via unified large-scale audio-language models,”arXiv preprint arXiv:2311.07919, 2023

    2023 arXiv

  2. [10]

    Qwen2-audio technical re- port,

    Y . Chu, J. Xu, Q. Yang, H. Wei, X. Wei, Z. Guo, Y . Leng, Y . Lv, J. He, J. Lin, C. Zhou, and J. Zhou, “Qwen2-audio technical re- port,”arXiv preprint arXiv:2407.10759, 2024

    2024 arXiv

  3. [11]

    Salmonn: Towards generic hearing abilities for large language models,

    C. Tang, W. Yu, G. Sun, X. Chen, T. Tan, W. Li, L. Lu, Z. Ma, and C. Zhang, “Salmonn: Towards generic hearing abilities for large language models,”arXiv preprint arXiv:2310.13289, 2023

    2023 arXiv

  4. [12]

    MERaLiON-AudioLLM: Technical report,

    Y . He, Z. Liu, S. Sun, B. Wang, W. Zhang, X. Zou, N. F. Chen, and A. T. Aw, “MERaLiON-AudioLLM: Technical report,”arXiv preprint arXiv:2412.09818, 2024

    2024 arXiv

  5. [13]

    Towards a speech foundation model for singapore and beyond,

    M. Huzaifah, T. Liu, H. B. Sailor, K. M. Tan, T. K. Vangani, Q. Wang, J. H. Wong, N. F. Chen, and A. T. Aw, “Towards a speech foundation model for singapore and beyond,”arXiv preprint arXiv:2412.11538, 2024

    2024 arXiv

  6. [14]

    BLSP-Emo: Towards empathetic large speech-language mod- els,

    C. Wang, M. Liao, Z. Huang, J. Wu, C. Zong, and J. Zhang, “BLSP-Emo: Towards empathetic large speech-language mod- els,”arXiv preprint arXiv:2406.03872, 2024

    2024 arXiv

  7. [15]

    AudioPaLM: A large language model that can speak and listen,

    P. K. Rubenstein, C. Asawaroengchai, D. D. Nguyen, A. Bapna, Z. Borsos, F. d. C. Quitry, P. Chen, D. E. Badawy, W. Han, E. Kharitonov, and other, “AudioPaLM: A large language model that can speak and listen,”arXiv preprint arXiv:2306.12925, 2023

    2023 arXiv

  8. [16]

    LauraGPT: Listen, attend, understand, and regenerate audio with GPT,

    Q. Chen, Y . Chu, Z. Gao, Z. Li, K. Hu, X. Zhou, J. Xu, Z. Ma, W. Wang, S. Zhenget al., “LauraGPT: Listen, attend, understand, and regenerate audio with GPT,”arXiv preprint arXiv:2310.04673, 2023

    2023 arXiv

  9. [17]

    Advancing large lan- guage models to capture varied speaking styles and respond prop- erly in spoken conversations,

    G.-T. Lin, C.-H. Chiang, and H.-Y . Lee, “Advancing large lan- guage models to capture varied speaking styles and respond prop- erly in spoken conversations,” inProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, 2024, pp. 6626–6642

    2024

  10. [18]

    BLSP: Bootstrapping language-speech pre-training via behavior alignment of continuation writing,

    C. Wang, M. Liao, Z. Huang, J. Lu, J. Wu, Y . Liu, C. Zong, and J. Zhang, “BLSP: Bootstrapping language-speech pre-training via behavior alignment of continuation writing,”arXiv preprint arXiv:2309.00916, 2024

    2024 arXiv

  11. [19]

    Paralinguistics-aware speech- empowered large language models for natural conversation,

    H. Kim, S. Seo, K. Jeong, O. Kwon, S. Kim, J. Kim, J. Lee, E. Song, M. Oh, J.-W. Haet al., “Paralinguistics-aware speech- empowered large language models for natural conversation,” in Neural Information Processing Systems (NeurIPS), 2024

    2024

  12. [20]

    Frozen large language mod- els can perceive paralinguistic aspects of speech,

    W. Kang, J. Jia, C. Wu, W. Zhou, E. Lakomkin, Y . Gaur, L. Sari, S. Kim, K. Li, J. Mahadeokaret al., “Frozen large language mod- els can perceive paralinguistic aspects of speech,”arXiv preprint arXiv:2410.01162, 2024

    2024 arXiv

  13. [21]

    AudioBench: A universal benchmark for audio large language models,

    B. Wang, X. Zou, G. Lin, S. Sun, Z. Liu, W. Zhang, Z. Liu, A. Aw, and N. F. Chen, “AudioBench: A universal benchmark for audio large language models,”NAACL, 2025

    2025

  14. [22]

    Dynamic-superb: To- wards a dynamic, collaborative, and comprehensive instruction- tuning benchmark for speech,

    C.-Y . Huang, K.-H. Lu, S.-H. Wang, C.-Y . Hsiao, C.-Y . Kuan, H. Wu, S. Arora, K.-W. Changet al., “Dynamic-superb: To- wards a dynamic, collaborative, and comprehensive instruction- tuning benchmark for speech,” inIEEE International Conference on Acoustics, Speech and Signal ...

    2024

  15. [23]

    AIR-bench: Benchmark- ing large audio-language models via generative comprehension,

    Q. Yang, J. Xu, W. Liu, Y . Chu, Z. Jiang, X. Zhou, Y . Leng, Y . Lv, Z. Zhao, C. Zhou, and J. Zhou, “AIR-bench: Benchmark- ing large audio-language models via generative comprehension,” inProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics,...

    2024

  16. [24]

    Listen, think, and understand,

    Y . Gong, H. Luo, A. H. Liu, L. Karlinsky, and J. Glass, “Listen, think, and understand,” inInternational Conference on Learning Representations (ICLR), 2024

    2024

  17. [25]

    MMAU: A mas- sive multi-task audio understanding and reasoning benchmark,

    S. Sakshi, U. Tyagi, S. Kumar, A. Seth, R. Selvakumar, O. Nieto, R. Duraiswami, S. Ghosh, and D. Manocha, “MMAU: A mas- sive multi-task audio understanding and reasoning benchmark,” inInternational Conference on Learning Representations (ICLR), 2025

    2025

  18. [26]

    IEMOCAP: interactive emotional dyadic motion capture database,

    C. Busso, M. Bulut, C.-C. Lee, E. A. Kazemzadeh, E. M. Provost, S. Kim, J. N. Chang, S. Lee, and S. S. Narayanan, “IEMOCAP: interactive emotional dyadic motion capture database,”Language Resources and Evaluation, vol. 42, pp. 335–359, 2008

    2008

  19. [27]

    MELD: A multimodal multi-party dataset for emo- tion recognition in conversations,

    S. Poria, D. Hazarika, N. Majumder, G. Naik, E. Cambria, and R. Mihalcea, “MELD: A multimodal multi-party dataset for emo- tion recognition in conversations,” inProceedings of the 57th An- nual Meeting of the Association for Computational Linguistics, 2019, pp. 527–536

    2019

  20. [28]

    What’s basic about basic emotions?

    A. Ortony and T. Turner, “What’s basic about basic emotions?” Psychological review, pp. 315–331, 1990

    1990

  21. [29]

    Theories of emotion,

    R. Plutchik and H. Kellerman, “Theories of emotion,”Academic Press, 2013

    2013

  22. [30]

    EmoBox: Multilingual multi-corpus speech emotion recognition toolkit and benchmark,

    Z. Ma, M. Chen, H. Zhang, Z. Zheng, W. Chen, X. Li, J. Ye, X. Chen, and T. Hain, “EmoBox: Multilingual multi-corpus speech emotion recognition toolkit and benchmark,” inProc. In- terspeech, 2024, pp. 1580–1584

    2024

  23. [31]

    Evidence for a three-factor the- ory of emotions,

    J. A. Russell and A. Mehrabian, “Evidence for a three-factor the- ory of emotions,”Journal of Research in Personality, vol. 11, no. 3, pp. 273–294, 1977

    1977

  24. [32]

    Goemotions: A dataset of fine-grained emo- tions,

    D. Demszky, D. Movshovitz-Attias, J. Ko, A. S. Cowen, G. Ne- made, and S. Ravi, “Goemotions: A dataset of fine-grained emo- tions,” inAnnual Meeting of the Association for Computational Linguistics, 2020

    2020

  25. [33]

    emotion2vec: Self-supervised pre-training for speech emotion representation,

    Z. Ma, Z. Zheng, J. Ye, J. Li, Z. Gao, S. Zhang, and X. Chen, “emotion2vec: Self-supervised pre-training for speech emotion representation,”Findings of the Association for Computational Linguistics (ACL), 2024

    2024

  26. [34]

    Dawn of the trans- former era in speech emotion recognition: Closing the valence gap,

    J. Wagner, A. Triantafyllopoulos, H. Wierstorf, M. Schmitt, F. Burkhardt, F. Eyben, and B. W. Schuller, “Dawn of the trans- former era in speech emotion recognition: Closing the valence gap,”IEEE Transactions on Pattern Analysis and Machine Intel- ligence, pp. 1–13, 2023

    2023

  27. [35]

    Building naturalistic emotionally bal- anced speech corpus by retrievingemotional speech from existing podcast recordings,

    R. Lotfian and C. Busso, “Building naturalistic emotionally bal- anced speech corpus by retrievingemotional speech from existing podcast recordings,”IEEE Transactions on Affective Computing, vol. 10, no. 4, pp. 471–483, 2019

    2019

  28. [36]

    WavLM: Large-scale self- supervised pre-training for full stack speech processing,

    S. Chen, C. Wang, Z. Chen, Y . Wu, S. Liu, Z. Chen, J. Li, N. Kanda, T. Yoshioka, X. Xiaoet al., “WavLM: Large-scale self- supervised pre-training for full stack speech processing,”IEEE Journal of Selected Topics in Signal Processing, vol. 16, no. 6, pp. 1505–1518, 2022

    2022

  29. [37]

    Ecapa-tdnn: Emphasized channel attention, propagation and aggregation in tdnn based speaker verification,

    B. Desplanques, J. Thienpondt, and K. Demuynck, “Ecapa-tdnn: Emphasized channel attention, propagation and aggregation in tdnn based speaker verification,” inProc. Interspeech, 2020, pp. 3830–3834

    2020

  30. [38]

    V oxCeleb2: Deep speaker recognition,

    J. S. Chung, A. Nagrani, and A. Zisserman, “V oxCeleb2: Deep speaker recognition,” inProc. Interspeech, 2018, pp. 1086–1090

    2018

  31. [39]

    WhisperX: Time- accurate speech transcription of long-form audio,

    M. Bain, J. Huh, T. Han, and A. Zisserman, “WhisperX: Time- accurate speech transcription of long-form audio,” inProc. Inter- speech, 2023

    2023

  32. [40]

    The Llama 3 herd of models,

    A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Let- man, A. Mathur, A. Schelten, A. Yang, A. Fanet al., “The Llama 3 herd of models,”CoRR, 2024

    2024

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.