REVIEW 5 cited by
MuChoMusic: Evaluating Music Understanding in Multimodal Audio-Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Multimodal models that jointly process audio and language hold great promise in audio understanding and are increasingly being adopted in the music domain. By allowing users to query via text and obtain information about a given audio input, these models have the potential to enable a variety of music understanding tasks via language-based interfaces. However, their evaluation poses considerable challenges, and it remains unclear how to effectively assess their ability to correctly interpret music-related inputs with current methods. Motivated by this, we introduce MuChoMusic, a benchmark for evaluating music understanding in multimodal language models focused on audio. MuChoMusic comprises 1,187 multiple-choice questions, all validated by human annotators, on 644 music tracks sourced from two publicly available music datasets, and covering a wide variety of genres. Questions in the benchmark are crafted to assess knowledge and reasoning abilities across several dimensions that cover fundamental musical concepts and their relation to cultural and functional contexts. Through the holistic analysis afforded by the benchmark, we evaluate five open-source models and identify several pitfalls, including an over-reliance on the language modality, pointing to a need for better multimodal integration. Data and code are open-sourced.
Forward citations
Cited by 5 Pith papers
-
Music I Care About: Automated Multimodal Benchmarking of LLM Music Perception Skills on (Almost) Any Music
A meta-benchmark that auto-generates multimodal music-perception multiple-choice tests from user symbolic music, demonstrated on ChoraleBricks with text-only and white-noise controls.
-
Benchmarking and Confidence Evaluation of LALMs For Temporal Reasoning
Open-source audio-language models perform far below humans on a new 600-item benchmark of temporal reasoning in sound, and their accuracy does not track a proposed perturbation-based uncertainty measure.
-
From Audio Deepfake Detection to AI-Generated Music Detection -- A Pathway and Overview
A review of AI-generated music detection that proposes intrinsic music features and multimodal fusion as the basis for adapting audio deepfake detection methods.
-
Breaking the Barriers of Text-Hungry and Audio-Deficient AI
A proposed audio-native translation framework called MAST with fractional diffusion is described, but no evidence is given that it produces working translations.
-
Align Anything: Training All-Modality Models to Follow Instructions with Language Feedback
The paper proposes learning from language feedback to synthesize multimodal preference pairs, but the evidence is weakened by an undefined improvement metric and small, unvalidated effect sizes.
Discussion (0). Continue with ORCID to comment.