Pith. sign in

REVIEW 3 major objections 6 minor 3 cited by

SAKURA: On the Multi-hop Reasoning of Large Audio-Language Models Based on Speech and Audio Information

T0 review · 3 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Even when large audio-language models accurately extract speech or audio attributes, they fail to carry those latent representations into multi-hop reasoning, a new benchmark shows.

desk verdict Useful new benchmark, but the headline claim about a modality-integration deficit is undercut by an oracle-caption control that doesn't isolate reasoning from perception. read the letter →

arxiv 2505.13237 v3 pith:GJMAZEHJ submitted 2025-05-19 eess.AS cs.CLcs.SD

Chih-Kai Yang , Neo Ho , Yen-Ting Piao , Hung-yi Lee This is my paper · ORCID
classification eess.AScs.CLcs.SD
keywords largeaudio-languagemodelmulti-hopreasoningspeechunderstandingaudiomultimodalbenchmarkattributeslatentrepresentations
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces SAKURA, a benchmark of 4,000 human-verified multiple-choice questions that tests large audio-language models (LALMs) on single-hop perception and multi-hop reasoning across four speech/audio attributes: speaker gender, spoken language, speaker emotion, and animal sound. Its central finding is that LALMs often answer single-hop attribute questions correctly but then fail on multi-hop questions that require reasoning over that same information. When the same attribute information is supplied as text instead of speech or audio, the same models reason much better. The paper interprets this gap as a lack of genuine multimodal integration: the models' reasoning remains text-driven even in end-to-end audio-language architectures.

What carries the argument

The central mechanism is the controlled single-hop/multi-hop comparison built into SAKURA. Each of the four tracks shares the same speech/audio inputs between single-hop and multi-hop sub-tracks, so multi-hop questions can be conditioned on whether the model perceived the attribute correctly. The decisive instrument is the text-based control condition: replacing the audio input with an oracle caption of the attribute isolates the reasoning step from perception, and the gap between speech/audio-based and text-based multi-hop accuracy on the same correctly-perceived instances is what carries the paper's claim.

What would settle it

Take any LALM that answers a SAKURA single-hop question correctly, then before the multi-hop question ask the model to state the attribute it heard (e.g., 'What animal is in the sound?'); if multi-hop accuracy on instances with correct self-reported attributes reaches the text-condition level, the integration deficit is in making the latent representation available to reasoning, not in the reasoning step itself. Conversely, if the model still fails despite correctly verbalizing the attribute, the deficit lies deeper in the reasoning module.

Watch

Extended reading notes

Core claim

The paper claims that current LALMs exhibit a modality-integration deficit rather than a pure perception deficit. In Section 4.2, the authors compare multi-hop accuracy on instances where each model already answered the corresponding single-hop question correctly, under two conditions: raw speech/audio input, and a textual caption stating the relevant attribute (e.g., 'The animal making the sound is cat.'). Across models, accuracy is consistently and often dramatically higher in the text condition; for instance, DeSTA2 rises from 65.8% to 93.3% average multi-hop accuracy, and Gemini-1.5-pro from 53.7% to 84.4%. Because the only intended difference is modality, the authors conclude that even when LALMs extract accurate information, they fail to incorporate latent speech/audio representations into the reasoning process, contradicting the expectation that end-to-end models unify understanding with reasoning.

Load-bearing premise

The argument assumes that replacing the audio with a textual caption changes only the input modality, and that a correct single-hop answer guarantees the model actually has the attribute in a usable form when it tackles the multi-hop question; if either assumption fails, the speech-versus-text gap could overstate a lack of true multimodal integration.

Editorial extensions

If this is right

  • Improving LALM perception alone will not fix multi-hop reasoning; the integration of extracted attributes into the reasoning path must be addressed separately.
  • Current LALMs fall short of cascaded systems such as ASR plus audio captioning plus an LLM, which outperforms most end-to-end LALMs on this benchmark.
  • Benchmark designers should treat single-hop accuracy as insufficient evidence that a model can use the perceived attribute in downstream reasoning.
  • The finding predicts that probing or prompting LALMs to verbalize the extracted attribute before answering a multi-hop question should close part of the gap, if the deficit is in integrating latent representations.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the integration deficit is real, a natural fix is to train LALMs on objectives that require reasoning over the audio encoder's latent state, not just the text decoder's output; this could be tested by fine-tuning with multi-hop audio examples and measuring whether the text-vs-audio gap shrinks.
  • A sharper test would compare not only whole-model accuracy but whether the attribute is linearly decodable from the model's internal representations at the multi-hop step; a decodability failure would mean the perception is not actually available to the reasoning module, while decodability with continued failure would pinpoint a reasoning-module limitation.
  • Because SAKURA covers only four attributes and relatively clean audio, extending it to noisy or overlapping acoustic conditions could reveal whether integration failures worsen when the extracted representation is weaker or noisier.
  • The cascade comparison hints that the text interface remains the most reliable carrier of attribute information; one could extend that by feeding ASR transcripts or captions from a stronger AAC model into LALMs to see whether the integration gap tracks the quality of the external text.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces SAKURA, a benchmark of 4000 human-verified multiple-choice questions for evaluating multi-hop reasoning in large audio-language models (LALMs) based on speech and audio. It comprises four attribute tracks (speaker gender, spoken language, speaker emotion, animal sound), each with single-hop perception and multi-hop reasoning sub-tracks that share the same audio inputs. The authors evaluate eight LALMs and two cascaded systems, report single-hop and multi-hop accuracies, and analyze a filtered subset where each model answered the corresponding single-hop question correctly, comparing speech/audio-based versus text-caption-based multi-hop reasoning. They conclude that LALMs fail to integrate extracted speech/audio information into reasoning even when extraction appears to succeed.

Significance. If the central claim holds, SAKURA is a valuable resource for the community: the dataset is human-verified with high agreement, audio sources are diverse and balanced, and the single-hop/multi-hop split is clean. The evaluation uses an LLM judge that was validated against human judgments on 200 samples with 99.5% agreement, and the benchmark and evaluation code are publicly released. The finding that current LALMs lag behind cascaded systems and behind their own text-based reasoning is a falsifiable, important observation that challenges the end-to-end design of LALMs and should spur research on multimodal integration. The comparison across open-source, proprietary, and cascaded systems also provides a useful reference point.

major comments (3)
  1. [Section 4.2, Table 3] The claim that 'even when extracting accurate information, LALMs fail to incorporate latent speech/audio representations into the reasoning process' is not fully supported by the experimental design. The text-based control condition provides an oracle caption (e.g., 'The animal making the sound is cat.'), which guarantees the attribute is available in clean propositional form, whereas the speech/audio condition's filtered subset is based only on the model's multiple-choice single-hop answer. For models with near-chance single-hop accuracy, such as GAMA-IT on Language (5.6% versus 25% chance), many 'correct' single-hop items are likely due to guessing or systematic biases, so the filtered S/A subset does not reliably isolate genuine extraction. Consequently, the S/A-versus-Text gap may partly reflect residual perception failures rather than a modality-integration deficit. The authors should either add a stricter extraction verification (e.g., open-ended single-hop questions, model-generated captions, or confidence-based filtering) or substantially soften the claim and limit it to models with high single-hop accuracy.
  2. [Table 3] The 95% confidence intervals in Table 3 appear to be computed over the full 500-item track rather than over the smaller filtered subset. For example, GAMA-IT's Language single-hop accuracy is 5.6%, giving roughly 28 filtered instances, but the table reports a CI of ±4.0 for the S/A multi-hop accuracy, which is implausible for n=28 (a binomial 95% CI would be approximately ±17). This overstates the precision of the estimates and may mislead readers about the statistical significance of the observed gaps. The authors should recompute all Table 3 CIs using the actual filtered sample sizes and report the per-cell n in the table or in a supplementary table.
  3. [Section 4.2] The statement that 'the only difference between these conditions is the modality used to present attribute information' is imprecise. The text captions differ not only in modality but also in information form: they are clean, propositional, and directly relevant, whereas the audio contains acoustic ambiguity and requires the model to internally derive the same proposition. Even when single-hop is answered correctly, the latent speech/audio representation may be less accessible to the reasoning modules than an explicit textual proposition. This does not invalidate the comparison, but it means the observed gap is an upper bound on a pure integration deficit rather than a direct measure of it. The paper should acknowledge this confound in the interpretation.
minor comments (6)
  1. [Table 2, caption] The omission of GPT-4o Audio's Gender track results is noted only in the main text; the table caption should also state that these entries are omitted due to the model's refusal to answer gender-related questions.
  2. [Section 2.1 and Table 2] The number of answer choices per track is not stated explicitly. The chance-level row implies four choices for Language, Emotion, and Animal, and two for Gender, but the paper describes eight languages in the Language track and five emotions in the Emotion track. Please clarify how choices are sampled and whether the number of choices is constant within each track.
  3. [Section 3.1] The LLM-as-a-judge evaluation would benefit from more detail: include the exact prompt or a reference to the appendix, and describe how the 200 samples for human verification were selected and how the 99.5% agreement was computed (e.g., Cohen's kappa or simple agreement).
  4. [Table 3, Average column] Please specify how the Average column is computed, in particular whether it is an unweighted mean of the four track accuracies or a pooled accuracy across tracks. This affects readability and comparability.
  5. [Throughout] There are minor typographical and formatting issues: 'benchmar k' in the SAKURA expansion, inconsistent use of 'GPT-4o Audio' versus 'gpt-4o-audio-preview', and the unusual rendering of 'S ¸as ¸maz' in the references. These should be cleaned up.
  6. [Section 1 and Conclusion] The claim of being the 'first benchmark' for this purpose should be qualified as 'to our knowledge' and should be checked against concurrent or recent benchmarks such as MMAU, which is cited but may overlap in reasoning-focused tasks.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: SAKURA is an externally anchored benchmark evaluation; its central claim is an empirical finding, not a quantity derived from its own definitions.

full rationale

The paper contains no derivation chain in which a target quantity is defined in terms of the quantity it is supposed to predict. SAKURA's speech/audio data come from external corpora (Common Voice, CREMA-D, MELD, ESC-50, and the Sasmaz et al. dataset); the question-answer pairs are generated by GPT-4o and human-verified with at least three annotators per item; and the LLM-as-a-judge evaluation is validated against human annotations on 200 samples with 99.5% agreement. Tables 2 and 3 report measured accuracies with confidence intervals, not fitted parameters later renamed as predictions. The central claim that LALMs fail to incorporate latent speech/audio representations into multi-hop reasoning despite accurate single-hop extraction is an interpretation of a controlled experimental comparison, and it remains testable against external model outputs. The text-based condition uses oracle captions, and the single-hop-correct filter may admit lucky guesses for low-accuracy models; these are potential validity threats to the experimental conclusion, but they are not cases where a predicted quantity equals an input by construction. Self-citations in the references are contextual prior work and are not load-bearing for the benchmark's construction or results. The paper is therefore not circular.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper introduces no fitted numeric parameters and no new physical or mechanistic entities. Its inferences rest on dataset label reliability, machine-generated but human-verified question content, and the LLM-judge agreement result. The most fragile premise is the controlled comparison in Section 4.2, where text-based inputs are oracle captions rather than model-extracted attributes.

assumptions (4)
  • domain assumption Labels in Common Voice, CREMA-D, MELD, ESC-50, and Sasmaz et al. are correct ground truth for the target attributes.
    The benchmark assigns golden answers from these external dataset labels without independent acoustic verification.
  • domain assumption GPT-4o-generated questions and distractors, after human consensus review, are objective, grounded, and free of ambiguity.
    Topic selection and candidate generation are machine-proposed; human annotators filtered by relevance, objectivity, and uniqueness criteria.
  • domain assumption The LLM-as-a-judge scoring by GPT-4o is a valid proxy for answer correctness.
    The paper reports 99.5% agreement with human judgments on 200 sampled responses, but not all judgments are human-checked.
  • domain assumption Correctly answering the single-hop question implies the model has the attribute available in a form usable for the multi-hop step.
    This is the load-bearing premise of the speech/audio versus text comparison in Section 4.2; the text control also supplies an oracle caption, so the comparison may overstate the integration deficit.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SAKURA: On the Multi-hop Reasoning of Large Audio-Language Models Based on Speech and Audio Information." pith.science (2026). https://pith.science/paper/GJMAZEHJ

@misc{pith2026250513237,
  author       = {Pith},
  title        = {Pith review of: SAKURA: On the Multi-hop Reasoning of Large Audio-Language Models Based on Speech and Audio Information},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GJMAZEHJ}},
  note         = {Machine review of arXiv:2505.13237}
}
read the original abstract

Large audio-language models (LALMs) extend the large language models with multimodal understanding in speech, audio, etc. While their performances on speech and audio-processing tasks are extensively studied, their reasoning abilities remain underexplored. Particularly, their multi-hop reasoning, the ability to recall and integrate multiple facts, lacks systematic evaluation. Existing benchmarks focus on general speech and audio-processing tasks, conversational abilities, and fairness but overlook this aspect. To bridge this gap, we introduce SAKURA, a benchmark assessing LALMs' multi-hop reasoning based on speech and audio information. Results show that LALMs struggle to integrate speech/audio representations for multi-hop reasoning, even when they extract the relevant information correctly, highlighting a fundamental challenge in multimodal reasoning. Our findings expose a critical limitation in LALMs, offering insights and resources for future research.

Figures

Figures reproduced from arXiv: 2505.13237 by the authors.

Figure 1
Figure 1. Overview of dataset construction and experiments. Question-answer pairs were generated by GPT-4o and manually verified. LALMs were evaluated on speech/audio-based vs. text-based reasoning, where text-based replaces auditory inputs with captions, assessing their ability to integrate auditory information. Icon source: https://www.flaticon.com trait that matches the identified animal). Each sub-track con￾tains 500 mult… view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Large Audio Language Models for Spoofing-Aware Speaker Verification

    cs.SD 2026-07 conditional novelty 6.0 of 10

    Adapted LALMs can reach competitive spoofing-aware speaker verification (89.3% accuracy, 0.19 min a-DCF on an ASVspoof5 subset), though zero-shot performance is near chance.

  2. Escaping the Procrustean Bed: Groupwise Orthogonal Connectors for Audio-Language Models

    cs.SD 2026-07 unverdicted novelty 6.0 of 10

    ORCA splits Q-Former queries into orthogonally constrained groups, reversing directional collapse and speaker-indistinguishability in audio-LLM connectors and gaining 26.4 points on SAKURA multi-hop reasoning.

  3. Reducing Object Hallucination in Large Audio-Language Models via Audio-Aware Decoding

    eess.AS 2025-06 conditional novelty 4.0 of 10

    Audio-Aware Decoding, a contrastive decoding method that uses silent audio as the no-context baseline, reduces object hallucination and improves accuracy across three large audio-language models.

Reference graph

Works this paper leans on

54 extracted references · 39 canonical work pages · cited by 3 Pith papers

  1. [1]

    What is the animal in the sound ?

    Introduction Large language models (LLMs) [1, 2] have revolutionized AI research, extending beyond natural language processing to domain-specific applications like computer vision [3] and speech processing [4, 5]. This shift drives the rise of large mul- timodal models, e.g., large vision-language models (LVLMs) [6,7] and large audio-language models (LALM...

    work page

  2. [2]

    Avg”, “Std

    Dataset construction 2.1. Overview SAKURA evaluates LALMs’ multi-hop reasoning with speech/audio information across four tracks, covering funda- mental attributes: speaker gender (Gender), spoken language (Language), speaker emotion (Emotion), and animal sounds (Animal), chosen for their significance in speech/audio process- ing. Each track includes singl...

    arXiv 2025

  3. [3]

    determining the age of the speaker

    Relevance and Groundedness: Topics must be relevant to the track’s attribute and supported by the corresponding in- formation. For example, topics like “determining the age of the speaker” in the Gender track or “determining the religious beliefs of the speaker” in the Language track were excluded

    work page

  4. [4]

    the association of animals and human personality

    Objectivity: Topics must be objective and unbiased. For ex- ample, “the association of animals and human personality” was excluded due to cultural ambiguity

    work page

  5. [5]

    feeding habits

    Uniqueness: Topics should be distinct. For example, only one of “feeding habits” and “food sources” could be included. For each topic, GPT-4o generated diverse questions along with golden answer candidates and distractors for all attribute labels, forming a choice pool for pairing with speech/audio data. For example, for a question on “physical traits” wi...

    work page 2024

  6. [6]

    Evaluation metrics Since SAKURA comprises multiple-choice questions, accu- racy is a natural metric

    Experimental setups 3.1. Evaluation metrics Since SAKURA comprises multiple-choice questions, accu- racy is a natural metric. However, because LALMs sometimes generate descriptive responses instead of explicitly selecting a choice, we adopt an LLM-as-a-judge [37] approach following prior works [22, 23]. Specifically, we employ GPT-4o5 as the evaluator. Du...

    work page

  7. [7]

    Each question in SAKURA has exactly one correct answer. If a model fails to select one and only one choice from the given choices, either by selecting multiple choices or failing to explicitly choose any, it should be marked as incorrect

    work page

  8. [8]

    The evaluator must assess the alignment between the golden 5gpt-4o-2024-11-20, https://openai.com answer and the model’s response, providing an explanation to enhance the evaluation transparency

    work page 2024

Show all 54 references
  1. [9]

    cor- rect/incorrect

    The final judgment should be summarized as a binary “cor- rect/incorrect” label to facilitate post-processing. We incorporate in-context examples to enhance evaluation quality. To verify the evaluator’s reliability, we conducted hu- man verification on its judgments on 200 ran...

    2024

  2. [10]

    The animal making the sound is cat

    Results 4.1. Main results on SAKURA Table 2 shows the baseline performances on SAKURA, with chance level included for reference. GPT-4o Audio’s results on the Gender track are omitted as it refuses to answer gender- related questions due to post-training constraints, consisten...

  3. [11]

    Our findings show that LALMs struggle to recognize certain speech and audio attributes, exhibiting perception blind spots

    Conclusion, limitations, and future work We introduce SAKURA, the first benchmark for systematically evaluating LALMs’ multi-hop reasoning with speech and audio. Our findings show that LALMs struggle to recognize certain speech and audio attributes, exhibiting perception blind...

  4. [12]

    Acknowledgement We acknowledge the computational and storage support pro- vided by the National Center for High-performance Comput- ing (NCHC) of the National Applied Research Laboratories (NARLabs) in Taiwan

  5. [13]

    The llama 3 herd of models,

    A. Dubey et al. , “The llama 3 herd of models,” arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  6. [14]

    Gpt-4o system card,

    O. (2024) et al. , “Gpt-4o system card,” arXiv preprint arXiv:2410.21276, 2024

    2024 arXiv

  7. [15]

    Vipergpt: Visual inference via python execution for reasoning,

    D. Sur ´ıs, S. Menon, and C. V ondrick, “Vipergpt: Visual inference via python execution for reasoning,” Proceedings of IEEE Inter- national Conference on Computer Vision (ICCV), 2023

    2023

  8. [16]

    Audiogpt: Understanding and generating speech, music, sound, and talking head,

    R. Huang et al., “Audiogpt: Understanding and generating speech, music, sound, and talking head,” inProceedings of the AAAI Con- ference on Artificial Intelligence, 2024, pp. 23 802–23 804

    2024

  9. [17]

    Speech-copilot: Leveraging large language models for speech processing via task decomposition, modular- ization, and program generation,

    C.-Y . Kuan et al. , “Speech-copilot: Leveraging large language models for speech processing via task decomposition, modular- ization, and program generation,” in2024 IEEE Spoken Language Technology Workshop (SLT). IEEE, 2024, pp. 1060–1067

    2024

  10. [18]

    Visual instruction tuning,

    H. Liu et al., “Visual instruction tuning,” in Advances in Neural Information Processing Systems, 2023, pp. 34 892–34 916

    2023

  11. [19]

    Gemini 1.5: Unlocking multimodal under- standing across millions of tokens of context,

    G. Team et al. , “Gemini 1.5: Unlocking multimodal under- standing across millions of tokens of context,” arXiv preprint arXiv:2403.05530, 2024

    2024 arXiv

  12. [20]

    Joint audio and speech understanding,

    Y . Gong et al., “Joint audio and speech understanding,” in 2023 IEEE Automatic Speech Recognition and Understanding Work- shop (ASRU), 2023

    2023

  13. [21]

    Gama: A large audio-language model with advanced audio understanding and complex reasoning abilities,

    S. Ghosh et al. , “Gama: A large audio-language model with advanced audio understanding and complex reasoning abilities,” arXiv preprint arXiv:2406.11768, 2024

    2024 arXiv

  14. [22]

    SALMONN: Towards generic hearing abilities for large language models,

    C. Tang et al., “SALMONN: Towards generic hearing abilities for large language models,” in The Twelfth International Conference on Learning Representations, 2024

    2024

  15. [23]

    Developing instruction-following speech lan- guage model without speech instruction-tuning data,

    K.-H. Lu et al. , “Developing instruction-following speech lan- guage model without speech instruction-tuning data,” arXiv preprint arXiv:2409.20007, 2024

    2024 arXiv

  16. [24]

    Qwen-audio: Advancing universal audio under- standing via unified large-scale audio-language models,

    Y . Chu et al., “Qwen-audio: Advancing universal audio under- standing via unified large-scale audio-language models,” arXiv preprint arXiv:2311.07919, 2023

    2023 arXiv

  17. [25]

    Qwen2-audio technical report,

    Y . Chu, J. Xu, Q. Yang, H. Wei, X. Wei et al. , “Qwen2-audio technical report,” arXiv preprint arXiv:2407.10759, 2024

    2024 arXiv

  18. [26]

    A peek into token bias: Large language models are not yet genuine reasoners,

    B. Jiang et al., “A peek into token bias: Large language models are not yet genuine reasoners,” in Proceedings of the 2024 Con- ference on Empirical Methods in Natural Language Processing , Nov. 2024, pp. 4722–4756

    2024

  19. [27]

    Large language models cannot self-correct rea- soning yet,

    J. Huang et al., “Large language models cannot self-correct rea- soning yet,” in The Twelfth International Conference on Learning Representations, 2024

    2024

  20. [28]

    Premise order matters in reasoning with large language models,

    X. Chen et al. , “Premise order matters in reasoning with large language models,” in Forty-first International Conference on Ma- chine Learning, 2024

    2024

  21. [29]

    Measuring and narrowing the compositionality gap in language models,

    O. Press et al., “Measuring and narrowing the compositionality gap in language models,” in Findings of the Association for Com- putational Linguistics: EMNLP 2023, Dec. 2023, pp. 5687–5711

    2023

  22. [30]

    Distributional reasoning in llms: Parallel reasoning processes in multi-hop reasoning,

    Y . Shalev et al. , “Distributional reasoning in llms: Parallel reasoning processes in multi-hop reasoning,” arXiv preprint arXiv:2406.13858, 2024

    2024 arXiv

  23. [31]

    Do large language models latently perform multi- hop reasoning?

    S. Yang et al., “Do large language models latently perform multi- hop reasoning?” in Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Aug. 2024, pp. 10 210–10 229

    2024

  24. [32]

    Hopping too late: Exploring the limitations of large language models on multi-hop queries,

    E. Biran et al., “Hopping too late: Exploring the limitations of large language models on multi-hop queries,” in Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Nov. 2024, pp. 14 113–14 130

    2024

  25. [33]

    Investigating multi-hop factual shortcuts in knowl- edge editing of large language models,

    T. Ju et al., “Investigating multi-hop factual shortcuts in knowl- edge editing of large language models,” in Proceedings of the 62nd Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), Aug. 2024, pp. 8987–9001

    2024

  26. [34]

    Dynamic-SUPERB phase-2: A collabora- tively expanding benchmark for measuring the capabilities of spo- ken language models with 180 tasks,

    C.-y. Huang et al. , “Dynamic-SUPERB phase-2: A collabora- tively expanding benchmark for measuring the capabilities of spo- ken language models with 180 tasks,” in The Thirteenth Interna- tional Conference on Learning Representations, 2025

    2025

  27. [35]

    AIR-bench: Benchmarking large audio-language models via generative comprehension,

    Q. Yang et al., “AIR-bench: Benchmarking large audio-language models via generative comprehension,” in Proceedings of the 62nd Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), Aug. 2024, pp. 1979–1998

    2024

  28. [36]

    Advancing large lan- guage models to capture varied speaking styles and respond prop- erly in spoken conversations,

    G.-T. Lin, C.-H. Chiang, and H.-y. Lee, “Advancing large lan- guage models to capture varied speaking styles and respond prop- erly in spoken conversations,” in Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Aug...

    2024

  29. [37]

    Sd-eval: A benchmark dataset for spoken dialogue understanding beyond words,

    J. AO et al., “Sd-eval: A benchmark dataset for spoken dialogue understanding beyond words,” inAdvances in Neural Information Processing Systems, vol. 37, 2024, pp. 56 898–56 918

    2024

  30. [38]

    Listen and speak fairly: a study on semantic gender bias in speech integrated large language models,

    Y .-C. Lin et al. , “Listen and speak fairly: a study on semantic gender bias in speech integrated large language models,” in 2024 IEEE Spoken Language Technology Workshop (SLT) , 2024, pp. 439–446

    2024

  31. [39]

    Spoken stereoset: on eval- uating social bias toward speaker in speech large language mod- els,

    Y .-C. Lin, W.-C. Chen, and H.-Y . Lee, “Spoken stereoset: on eval- uating social bias toward speaker in speech large language mod- els,” in2024 IEEE Spoken Language Technology Workshop (SLT), 2024, pp. 871–878

    2024

  32. [40]

    Compa: Addressing the gap in compositional reasoning in audio-language models,

    S. Ghosh et al., “Compa: Addressing the gap in compositional reasoning in audio-language models,” in The Twelfth Interna- tional Conference on Learning Representations, 2024

    2024

  33. [41]

    MMAU: A massive multi-task audio understand- ing and reasoning benchmark,

    S. Sakshi et al., “MMAU: A massive multi-task audio understand- ing and reasoning benchmark,” in The Thirteenth International Conference on Learning Representations, 2025

    2025

  34. [42]

    Understanding sounds, missing the questions: The challenge of object hallucination in large audio-language models,

    C.-Y . Kuan et al., “Understanding sounds, missing the questions: The challenge of object hallucination in large audio-language models,” in Interspeech 2024, 2024, pp. 4144–4148

    2024

  35. [43]

    Common voice: A massively-multilingual speech corpus,

    R. Ardila et al. , “Common voice: A massively-multilingual speech corpus,” in Proceedings of the 12th Conference on Lan- guage Resources and Evaluation (LREC 2020) , 2020, pp. 4211– 4215

    2020

  36. [44]

    Crema-d: Crowd-sourced emotional multimodal actors dataset,

    H. Cao et al., “Crema-d: Crowd-sourced emotional multimodal actors dataset,” IEEE transactions on affective computing, vol. 5, no. 4, pp. 377–390, 2014

    2014

  37. [45]

    MELD: A multimodal multi-party dataset for emotion recognition in conversations,

    S. Poria et al. , “MELD: A multimodal multi-party dataset for emotion recognition in conversations,” inProceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Jul. 2019, pp. 527–536

    2019

  38. [46]

    Emotion detection on tv show transcripts with sequence-based convolutional neural networks,

    S. M. Zahiri et al., “Emotion detection on tv show transcripts with sequence-based convolutional neural networks,” in Workshops at the thirty-second aaai conference on artificial intelligence, 2018

    2018

  39. [47]

    Esc: Dataset for environmental sound classifica- tion,

    K. J. Piczak, “Esc: Dataset for environmental sound classifica- tion,” in Proceedings of the 23rd ACM international conference on Multimedia, 2015, pp. 1015–1018

    2015

  40. [48]

    Animal sound classification using a convolutional neural network,

    E. S ¸as ¸maz and F. B. Tek, “Animal sound classification using a convolutional neural network,” in 2018 3rd International Confer- ence on Computer Science and Engineering (UBMK) . IEEE, 2018, pp. 625–629

    2018

  41. [49]

    A survey on llm-as-a-judge,

    J. Gu et al. , “A survey on llm-as-a-judge,” arXiv preprint arXiv:2411.15594, 2024

    2024 arXiv

  42. [50]

    Robust speech recognition via large-scale weak supervision,

    A. Radford et al., “Robust speech recognition via large-scale weak supervision,” in International Conference on Machine Learning . PMLR, 2023, pp. 28 492–28 518

    2023

  43. [51]

    A preliminary exploration with gpt-4o voice mode,

    Y .-X. Lin, C.-K. Yang, W.-C. Chen, C.-A. Li, C.-y. Huang, X. Chen, and H.-y. Lee, “A preliminary exploration with gpt-4o voice mode,” arXiv preprint arXiv:2502.09940, 2025

    2025 arXiv

  44. [52]

    Llama-omni: Seamless speech interaction with large language models,

    Q. Fang et al., “Llama-omni: Seamless speech interaction with large language models,” arXiv preprint arXiv:2409.06666, 2024

    2024 arXiv

  45. [53]

    Moshi: a speech-text foundation model for real-time dialogue,

    A. D ´efossez et al., “Moshi: a speech-text foundation model for real-time dialogue,” arXiv preprint arXiv:2410.00037, 2024

    2024 arXiv

  46. [54]

    Building a taiwanese mandarin spoken lan- guage model: A first attempt,

    C.-K. Yang et al., “Building a taiwanese mandarin spoken lan- guage model: A first attempt,” arXiv preprint arXiv:2411.07111, 2024

    2024 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.