SAKURA: On the Multi-hop Reasoning of Large Audio-Language Models Based on Speech and Audio Information
Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 reserved pith:GJMAZEHJrecord.jsonopen to challenge →
read the original abstract
Large audio-language models (LALMs) extend the large language models with multimodal understanding in speech, audio, etc. While their performances on speech and audio-processing tasks are extensively studied, their reasoning abilities remain underexplored. Particularly, their multi-hop reasoning, the ability to recall and integrate multiple facts, lacks systematic evaluation. Existing benchmarks focus on general speech and audio-processing tasks, conversational abilities, and fairness but overlook this aspect. To bridge this gap, we introduce SAKURA, a benchmark assessing LALMs' multi-hop reasoning based on speech and audio information. Results show that LALMs struggle to integrate speech/audio representations for multi-hop reasoning, even when they extract the relevant information correctly, highlighting a fundamental challenge in multimodal reasoning. Our findings expose a critical limitation in LALMs, offering insights and resources for future research.
This paper has not been read by Pith yet.
Forward citations
Cited by 4 Pith papers
-
From Sounds to Scenes: A Benchmark for Evaluating Context-Aware Auditory Scene Understanding in Large Audio Language Models
Introduces CASU benchmark with four tasks to evaluate context-aware auditory scene understanding in LALMs via semi-synthetic audio compositions of speech, events, and environments.
-
Escaping the Procrustean Bed: Groupwise Orthogonal Connectors for Audio-Language Models
ORCA splits Q-Former queries into orthogonally constrained groups, reversing directional collapse and speaker-indistinguishability in audio-LLM connectors and gaining 26.4 points on SAKURA multi-hop reasoning.
-
A Survey of Large Audio Language Models: Generalization, Trustworthiness, and Outlook
A survey of Large Audio Language Models that establishes a taxonomy of trustworthiness vulnerabilities and proposes a Defense-in-Depth roadmap for audio intelligence.
-
A Survey of Audio Reasoning in Multimodal Foundation Models
A survey that provides a unified formulation of audio reasoning and reviews advances across Audio-to-Text, Audio-to-Speech, Audio-Visual, and Agentic paradigms while discussing challenges and future directions.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.