Pith. sign in

REVIEW 4 major objections 6 minor 68 references

CMI-Bench: A Comprehensive Benchmark for Evaluating Music Instruction Following

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read CMI-Bench recasts 14 MIR tasks as instruction-following prompts and shows that 11 open-source audio-text LLMs fall far behind supervised models on standard music metrics, except in captioning.

desk verdict Useful benchmark with a real comparability caveat: the multi-label embedding scoring breaks direct comparison with supervised baselines, but the core finding that LLMs lag far behind on most MIR tasks is solid. read the letter →

arxiv 2506.12285 v2 pith:RMY7VDBX submitted 2025-06-14 eess.AS cs.AIcs.LGcs.SD

Yinghao Ma , Siyou Li , Juntao Yu , Emmanouil Benetos , Akira Maezawa This is my paper
classification eess.AScs.AIcs.LGcs.SD
keywords musicinstructionfollowingaudio-textlargelanguagemodelsinformationretrievalbenchmarkMIRevaluationmetricsmulti-labeltaggingsequentialanalysisculturalandgenderbiasLLM
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

CMI-Bench reformulates the annotations of 20 music information retrieval (MIR) datasets into instruction-following prompts, covering 14 tasks from key detection and emotion regression to beat tracking and lyrics transcription. The paper's central claim is that this benchmark, scored with the same metrics used for supervised MIR systems, lets audio-text LLMs be compared directly against task-specific models, and that on this comparison 11 open-source audio-text LLMs fall far short of supervised systems on nearly every task. The main exception is music captioning, where LLMs perform competitively. The results also show that the best per-dataset scores tend to come from models whose training data overlaps the test set, and that fine-grained analysis reveals cultural, chronological, and gender bias.

What carries the argument

The carrying device is a self-instruction reformulation of MIR annotations: each dataset's labels become an instruction, an audio placeholder, and a few-shot example, covering multi-class, multi-label, regression, and sequential tuple outputs. The companion evaluation pipeline post-processes free-form LLM text into the input formats of standard MIR scoring, using the BGE text-embedding model's cosine similarity to convert free-form tag responses into soft tag sets before computing ROC-AUC and PR-AUC, and using mir_eval metrics for key, beat, melody, and lyrics tasks. This pipeline is what makes the claim of direct comparability with supervised MIR models operational.

What would settle it

Re-score the evaluated LLMs' multi-label outputs on one or two tagging datasets using strict matching to a fixed tag vocabulary instead of BGE cosine similarity, or run the supervised baselines through the same BGE-based scoring; if the performance gap on that dataset shrinks or disappears, the headline underperformance result is partly an artifact of the scoring protocol.

Watch

Extended reading notes

Core claim

The paper claims that a unified instruction-following benchmark for music is achievable by reinterpreting existing MIR annotations as natural-language instruction–response pairs and by feeding LLM outputs through a post-processing pipeline that converts free-form text into the exact formats expected by standard MIR metrics such as those in mir_eval. Evaluated on this benchmark, all 11 open-source audio-text LLMs studied perform significantly worse than task-specific supervised MIR models on classification, regression, multi-label tagging, and sequential tasks such as beat tracking, melody extraction, and playing-technique detection, with emotion regression and lyrics transcription close to unusable. The exception is music captioning, where LLMs are competitive. The paper further claims that peak performance on each task is typically reached by models trained on data overlapping the test set, evidence of limited generalization, and that per-tag and per-genre breakdowns show consistent bias toward Western instruments, mainstream genres, and particular voice types.

Load-bearing premise

The comparison assumes that scoring free-form LLM answers by embedding-based semantic similarity to tags preserves the same meaning as the exact tag scoring used to produce the supervised baselines, so the two sets of numbers belong on the same leaderboard.

Editorial extensions

If this is right

  • Any future audio-text LLM can now be ranked against traditional MIR systems on the same metrics, making cross-paradigm comparison a routine evaluation step.
  • Current open-source music LLMs are not ready to replace task-specific MIR models for classification, regression, or temporally structured analyses, though captioning is a plausible use case.
  • Sequential MIR tasks remain essentially unsolved by LLMs, pointing to timestamped pretraining and output-format supervision as the likely missing ingredients.
  • Reported gains from music-instruction models should be read with caution, since the best scores cluster on datasets already present in the models' training corpora.
  • Fine-grained scoring of instruments, genres, and voice tags can expose training-data bias, giving developers a concrete target for fairness work.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The BGE-based soft scoring could be tested against a strict-vocabulary re-scoring to verify whether the LLM-versus-supervised gap persists when synonyms are not rewarded; this is a check the paper does not run.
  • The same protocol could be applied to proprietary models such as GPT-4o and Gemini, which the paper mentions but does not evaluate, to see whether the gap is an open-source limitation or a general one.
  • The observed failure mode of copying few-shot examples verbatim in downbeat tracking suggests a useful new diagnostic: measuring output diversity and exact-copy rates alongside task accuracy.
  • Expanding the benchmark beyond Western, English-language music — the paper's own ethics statement acknowledges this skew — would directly test whether the cultural and gender biases it reports are fixed properties of current models or artifacts of training data.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces CMI-Bench, a benchmark for evaluating audio-text large language models (LLMs) on music instruction following across 14 MIR tasks and 20 dataset-task combinations. The authors reformulate existing MIR annotations into instruction-following prompts, release an evaluation toolkit, and benchmark 11 open-source audio-text LLMs. They report that LLMs fall significantly short of task-specific supervised MIR systems on most tasks, and they analyze cultural, chronological, and gender biases. The paper's central claim is that the benchmark adopts standardized evaluation metrics consistent with prior MIR literature, ensuring direct comparability with supervised approaches.

Significance. If the comparability issues are resolved, CMI-Bench would be a valuable community resource: it covers a broader task range than prior music LLM benchmarks, includes sequential tasks, releases code and test audio, and provides a reproducible evaluation harness. The bias analysis is a useful contribution, and the ethics statement's acknowledgment of the dataset's Western/English skew is a strength. However, the central claim of direct comparability is currently weakened by protocol mismatches in the multi-label and regression evaluations, so the headline quantitative comparisons need revision before the paper can serve as the unified benchmark it aims to be.

major comments (4)
  1. [§4.1.1, §E.3, Table 3 (MTT, M-G, M-E, M-I, M-50 rows)] The abstract and Section 1 promise "standardized evaluation metrics consistent with previous state-of-the-art MIR models, ensuring direct comparability." For the five multi-label datasets, however, LLM scores are computed via BGE embedding cosine similarity between predicted and ground-truth tag sets, while the SOTA numbers in Table 3 are per-tag ROC-AUC/PR-AUC computed from supervised probability outputs. The paper does not specify how a set-level cosine similarity is converted into per-instance, per-tag scores for ROC/PR, and no SOTA baseline is recomputed under the BGE protocol. Figure 5 shows that the choice of embedding method changes LLM scores substantially. Consequently, the headline numeric gap on these datasets is not a direct comparison but is protocol-dependent. The authors should either adopt exact-match tag scoring for both LLMs and SOTA, or recompute SOTA baselines under the same embedding protocol and report both sets of numbers.
  2. [§4.1.2, Table 3 EMO rows] The emotion regression protocol applies flooring to integers, z-score normalization of predictions, and mean-imputation for invalid outputs before computing R2. These choices are not standard in the supervised MIR literature used for the SOTA baselines, and the paper itself states in §5.1.6 that mean imputation often yields better R2 than the models' raw outputs. The reported R2 values therefore do not measure model performance under a fixed, comparable protocol; they partly measure the post-processing decisions. The authors should report raw-scale R2 (or correlation) with a clearly specified handling of invalid outputs, and justify why z-score normalization is applied to predictions but not to annotations (or state if both are normalized).
  3. [Table 3, SDD row] The SOTA ROUGE value for SDD is reported as 111.9, which exceeds the theoretical maximum of 100 for ROUGE. This indicates a typo or a different metric is being reported. Since the captioning comparison is used to show that LLMs are competitive on captioning, this error must be corrected and the metric definition clarified. The authors should also verify that the SOTA captioning numbers were computed on the same test split and with the same reference captions.
  4. [§5.1.1, Table 3] The provenance of the SOTA numbers in Table 3 is not described. For a benchmark whose core promise is direct comparability, the authors should specify for each SOTA system the exact checkpoint, input preprocessing, and evaluation script used, and confirm that the CMI-Bench test split matches the split used in the cited papers. Without this, the comparison cannot be reproduced or audited. Providing a reproducibility script for all SOTA numbers would strengthen the paper.
minor comments (6)
  1. [Abstract and Table 1] The abstract says "20 diverse datasets" but Table 1 contains 20 dataset-task rows over 18 unique datasets; please rephrase to "20 dataset-task combinations" or "18 datasets."
  2. [Appendix E.3] Appendix E.3 refers to "Table X" without a number; the placeholder should be resolved or the table should be included.
  3. [Table 2] Table 2 header contains "SALAMONN-Audio" instead of "SALMONN-Audio."
  4. [Section 3.2] Section 3.2 says "Guzheng (traditional Chinese Kyoto)"; this should be "Guzheng (traditional Chinese zither)."
  5. [Header license and Ethics Statement] The header license (CC BY 4.0) conflicts with the ethics statement (CC BY-NC-SA); please reconcile the two statements.
  6. [Introduction] The introduction says "as illustrated in Figure 5" for the instruction-following format, but Figure 5 in the appendix concerns multi-label evaluation methods; the intended reference seems to be Figure 1.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: CMI-Bench is an empirical benchmark with external metrics and published baselines; the BGE multi-label protocol is a validity/comparability concern, not a circular reduction.

full rationale

CMI-Bench does not derive predictions from fitted inputs; it repurposes existing MIR annotations into prompts and measures LLM outputs with externally defined metrics (mir_eval key/beat/melody scores, BLEU/METEOR/ROUGE/BERTScore, WER/CER, R2). The SOTA baselines in Table 3 come from prior published systems (MULAN, MERT, MARBLE, BeatNet, etc.), not from fitting the benchmark's own labels, and although several of these prior works overlap with the present authors, those citations are not load-bearing: the qualitative LLM-versus-supervised gap is consistent across many datasets whose SOTA sources are independent (e.g., MULAN for MTT, BeatNet for ballroom, HKDSME for MedleyDB). No benchmark parameter is fitted to the test set and then reported as a prediction. The embedding-based multi-label scoring (BGE cosine similarity, Sec. 4.1.1 and Appendix E.3) is a genuine protocol-comparability limitation: the supervised ROC/PR numbers were computed from per-tag probabilities, not from set-level cosine similarities, so the headline gap on those five datasets is partly protocol-dependent. However, this is a measurement-validity and reproducibility concern, not a circularity: LLM scores are not equal, by construction, to the supervised scores, nor to any fitted input. The paper's own ablation (Fig. 5) shows that the choice of scorer changes LLM numbers, which is evidence of protocol sensitivity rather than a self-fulfilling benchmark. Under the rule that a benchmark evaluated against external, code-released baselines should receive a low circularity score, the appropriate finding is no significant circularity.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The benchmark introduces no new physical or mathematical entities. The free parameters are all evaluation-protocol choices: embedding model for multi-label scoring, z-score normalization, mean replacement for invalid regression outputs, and metric tolerances. The axioms are domain assumptions about metric validity and baseline comparability. The paper's contribution is an evaluation artifact, so the ledger is dominated by evaluation design choices rather than scientific postulates.

free parameters (5)
  • BGE embedding model choice for multi-label scoring = BGE (bge-base, 330M)
    The choice of embedding model and the threshold/similarity mapping for converting free-form model outputs into tag sets is a hand-selected evaluation design decision. Appendix E.3 shows different embedding models (exact match, BERTScore, BGE, GTE-Qwen) give different scores, and the authors pick BGE as a compromise. This choice affects all ROC-AUC and PR-AUC numbers for multi-label tasks.
  • Z-score normalization of emotion regression outputs = Zero mean, unit variance
    The paper normalizes model outputs and annotations to zero mean and unit variance before computing R2. This choice is reasonable for comparing scales, but it changes the interpretation of R2 and can mask systematic biases in the model's scale usage.
  • Mean replacement for invalid emotion outputs = Dataset mean
    When a model returns no valid value, the paper assigns the dataset mean, which directly affects R2 scores for emotion regression. The paper notes this often improves R2 relative to the model's own outputs.
  • F-measure tolerance for beat tracking = 20 ms
    The 20 ms tolerance is a standard mir_eval setting, but it is a chosen evaluation parameter that affects all beat and downbeat F-measure scores.
  • Melody evaluation tolerance = 50 cents
    The 50 cents tolerance for melody accuracy is a standard mir_eval choice but is a parameter of the evaluation.
assumptions (4)
  • domain assumption The BGE embedding model accurately captures semantic equivalence between model predictions and ground-truth tags.
    E.3 uses BGE cosine similarity to score free-form multi-label outputs. If the embedding model does not align with human judgment of tag equivalence, the ROC-AUC and PR-AUC scores for tagging tasks are distorted.
  • domain assumption Standard MIR metrics (mir_eval) are valid for evaluating LLM outputs after post-processing.
    The paper assumes that reformatting LLM free-form outputs into timestamp lists and label sets preserves the semantics needed for mir_eval metrics. The appendix cases show many LLM outputs are structurally invalid and filtered, so the metric often measures the post-processing pipeline as much as the model.
  • domain assumption The SOTA baselines cited from prior work are computed on the same test splits as CMI-Bench.
    The paper compares against numbers from publications like [49]-[63], but it does not demonstrate that those baselines used the identical splits, audio preprocessing, and evaluation protocol. If splits differ, the reported gaps are partly an artifact.
  • domain assumption Music instruction following can be decomposed into the 14 selected MIR tasks.
    The benchmark claims comprehensiveness, but the selection omits many MIR tasks such as tempo estimation, chord recognition, source separation, and structural segmentation. Comprehensiveness is asserted relative to the included task set.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CMI-Bench: A Comprehensive Benchmark for Evaluating Music Instruction Following." pith.science (2026). https://pith.science/paper/RMY7VDBX

@misc{pith2026250612285,
  author       = {Pith},
  title        = {Pith review of: CMI-Bench: A Comprehensive Benchmark for Evaluating Music Instruction Following},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RMY7VDBX}},
  note         = {Machine review of arXiv:2506.12285}
}
read the original abstract

Recent advances in audio-text large language models (LLMs) have opened new possibilities for music understanding and generation. However, existing benchmarks are limited in scope, often relying on simplified tasks or multi-choice evaluations that fail to reflect the complexity of real-world music analysis. We reinterpret a broad range of traditional MIR annotations as instruction-following formats and introduce CMI-Bench, a comprehensive music instruction following benchmark designed to evaluate audio-text LLMs on a diverse set of music information retrieval (MIR) tasks. These include genre classification, emotion regression, emotion tagging, instrument classification, pitch estimation, key detection, lyrics transcription, melody extraction, vocal technique recognition, instrument performance technique detection, music tagging, music captioning, and (down)beat tracking: reflecting core challenges in MIR research. Unlike previous benchmarks, CMI-Bench adopts standardized evaluation metrics consistent with previous state-of-the-art MIR models, ensuring direct comparability with supervised approaches. We provide an evaluation toolkit supporting all open-source audio-textual LLMs, including LTU, Qwen-audio, SALMONN, MusiLingo, etc. Experiment results reveal significant performance gaps between LLMs and supervised models, along with their culture, chronological and gender bias, highlighting the potential and limitations of current models in addressing MIR tasks. CMI-Bench establishes a unified foundation for evaluating music instruction following, driving progress in music-aware LLMs.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

68 extracted references · 50 canonical work pages

  1. [1]

    A survey of large language models,

    W. X. Zhao, K. Zhou, J. Li, T. Tang, X. Wang, Y . Hou, Y . Min, B. Zhang, J. Zhang, Z. Dong et al., “A survey of large language models,” arXiv preprint arXiv:2303.18223, vol. 1, no. 2, 2023

    arXiv 2023

  2. [2]

    Foundation models for music: A survey,

    Y . Ma, A. Øland, A. Ragni, B. M. Del Sette, C. Saitis, C. Donahue, C. Lin, C. Plachouras, E. Benetos, E. Shatri et al., “Foundation models for music: A survey,” arXiv preprint arXiv:2408.14340, 2024

    arXiv 2024

  3. [3]

    Bb major

    CMI-BENCHMARK With CMI-Bench, we aim to address the following limi- tations in evaluating music understanding capabilities of audio-text LLMs. Previous benchmarks often cover only a narrow range of tasks, and no benchmark supports se- quential tasks, overlooking many classic challenges which are central to MIR research. Moreover, evaluation proto- cols ar...

    work page

  4. [4]

    lyrics is as fol- lows:

    EXPERIMENTS 4.1 Evaluation Protocol To enable rigorous and fair comparison with traditional MIR systems, we design an evaluation pipeline that closely follows the original task definitions and metrics. All model outputs are automatically post-processed to conform to each task’s expected format, ensuring compatibility with MIR evaluation tools such as mir_...

    work page

  5. [5]

    <|pitch|><|midi_pitch|>piano

    RESULTS AND DISCUSSION 5.1 Benchmarking Results Experiment results reveal several important observations about the current state of audio-text LLMs on MIR tasks. 5.1.1 LLMs Underperform Traditional MIR Baselines. Despite LLMs have achieved excellent results on music cap- tions and multi-choices QA, [4, 8–10, 37], all models in our study fall significantly...

    work page

  6. [6]

    Our results highlight a significant performance gap between LLMs and supervised MIR systems, with best models like Qwen2-Audio and GAMA also struggling with general- ization

    CONCLUSION We introduce CMI-Bench, a comprehensive benchmark for evaluating audio-text LLMs across diverse MIR tasks. Our results highlight a significant performance gap between LLMs and supervised MIR systems, with best models like Qwen2-Audio and GAMA also struggling with general- ization. Sequence-based tasks, such as melody extraction and beat trackin...

    work page

  7. [7]

    Music un- derstanding llama: Advancing text-to-music generation with question answering and captioning,

    S. Liu, A. S. Hussain, C. Sun, and Y . Shan, “Music un- derstanding llama: Advancing text-to-music generation with question answering and captioning,” in ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024, pp. 286–290

    work page 2024

  8. [8]

    Musilingo: Bridging music and text with pre-trained language models for music captioning and query response,

    Z. Deng, Y . Ma, Y . Liu, R. Guo, G. Zhang, W. Chen, W. Huang, and E. Benetos, “Musilingo: Bridging music and text with pre-trained language models for music captioning and query response,” in NAACL-HLT (Find- ings), 2024

    work page 2024

Show all 68 references
  1. [9]

    Qwen-audio: Advancing univer- sal audio understanding via unified large-scale audio- language models,

    Y . Chu, J. Xu, X. Zhou, Q. Yang, S. Zhang, Z. Yan, C. Zhou, and J. Zhou, “Qwen-audio: Advancing univer- sal audio understanding via unified large-scale audio- language models,” arXiv preprint arXiv:2311.07919 , 2023

    2023 arXiv

  2. [10]

    Salmonn: Towards generic hearing abilities for large language models,

    C. Tang, W. Yu, G. Sun, X. Chen, T. Tan, W. Li, L. Lu, M. Zejun, and C. Zhang, “Salmonn: Towards generic hearing abilities for large language models,” in The Twelfth International Conference on Learning Repre- sentations

  3. [11]

    Air-bench: Benchmarking large audio-language models via genera- tive comprehension,

    Q. Yang, J. Xu, W. Liu, Y . Chu, Z. Jiang, X. Zhou, Y . Leng, Y . Lv, Z. Zhao, C. Zhou et al., “Air-bench: Benchmarking large audio-language models via genera- tive comprehension,” in ACL (1), 2024

    2024

  4. [12]

    Muchomusic: Evaluating music understanding in multimodal audio-language models,

    B. Weck, I. Manco, E. Benetos, E. Quinton, G. Fazekas, and D. Bogdanov, “Muchomusic: Evaluating music understanding in multimodal audio-language models,” in Proceedings of the 25th International Society for Music Information Retrieval Conference, ISMIR 2024, San Francisco, Cal...

    2024 doi

  5. [13]

    Omnibench: Towards the future of universal omni-language models,

    Y . Li, G. Zhang, Y . Ma, R. Yuan, K. Zhu, H. Guo, Y . Liang, J. Liu, Z. Wang, J. Yanget al., “Omnibench: Towards the future of universal omni-language models,” arXiv preprint arXiv:2409.15272, 2024

    2024

  6. [14]

    Mmau: A massive multi-task audio understanding and reasoning benchmark,

    S. Sakshi, U. Tyagi, S. Kumar, A. Seth, R. Selvakumar, O. Nieto, R. Duraiswami, S. Ghosh, and D. Manocha, “Mmau: A massive multi-task audio understanding and reasoning benchmark,” in The Thirteenth International Conference on Learning Representations

  7. [15]

    Super- naturalinstructions: Generalization via declarative in- structions on 1600+ nlp tasks,

    Y . Wang, S. Mishra, P. Alipoormolabashi, Y . Ko- rdi, A. Mirzaei, A. Naik, A. Ashok, A. S. Dhanasekaran, A. Arunkumar, D. Stap et al., “Super- naturalinstructions: Generalization via declarative in- structions on 1600+ nlp tasks,” in Proceedings of the 2022 Conference on Empi...

    2022

  8. [16]

    Self-instruct: Aligning language models with self-generated instructions,

    Y . Wang, Y . Kordi, S. Mishra, A. Liu, N. A. Smith, D. Khashabi, and H. Hajishirzi, “Self-instruct: Aligning language models with self-generated instructions,” in Proceedings of the 61st Annual Meeting of the Associ- ation for Computational Linguistics (Volume 1: Long Papers)...

    2023

  9. [17]

    Training language models to follow instructions with human feedback,

    L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray et al., “Training language models to follow instructions with human feedback,” Advances in neural information processing systems, vol. 35, pp. 27 730–27 744, 2022

    2022

  10. [18]

    Audio-flan: A prelim- inary release,

    L. Xue, Z. Zhou, J. Pan, Z. Li, S. Fan, Y . Ma, S. Cheng, D. Yang, H. Guo, Y . Xiaoet al., “Audio-flan: A prelim- inary release,” arXiv preprint arXiv:2502.16584, 2025

    2025 arXiv

  11. [19]

    The music maestro or the musically chal- lenged, a massive music evaluation benchmark for large language models,

    J. Li, L. Yang, M. Tang, C. Chenchong, Z. Li, P. Wang, and H. Zhao, “The music maestro or the musically chal- lenged, a massive music evaluation benchmark for large language models,” in Findings of the Association for Computational Linguistics ACL 2024, 2024, pp. 3246– 3257

    2024

  12. [20]

    Audiobench: A univer- sal benchmark for audio large language models,

    B. Wang, X. Zou, G. Lin, S. Sun, Z. Liu, W. Zhang, Z. Liu, A. Aw, and N. F. Chen, “Audiobench: A univer- sal benchmark for audio large language models,” arXiv preprint arXiv:2406.16020, 2024

    2024 arXiv

  13. [21]

    Mustango: Toward con- trollable text-to-music generation,

    J. Melechovsky, Z. Guo, D. Ghosal, N. Majumder, D. Herremans, and S. Poria, “Mustango: Toward con- trollable text-to-music generation,” in Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolo...

    2024

  14. [22]

    Muchin: a chinese colloquial description benchmark for evaluating language models in the field of music,

    Z. Wang, S. Li, T. Zhang, Q. Wang, P. Yu, J. Luo, Y . Liu, M. Xi, and K. Zhang, “Muchin: a chinese colloquial description benchmark for evaluating language models in the field of music,” inProceedings of the Thirty-Third International Joint Conference on Artificial Intelligenc...

    2024

  15. [23]

    Listen, think, and understand,

    Y . Gong, H. Luo, A. H. Liu, L. Karlinsky, and J. Glass, “Listen, think, and understand,” in International Confer- ence on Learning Representations, 2024

    2024

  16. [24]

    Joint audio and speech understanding,

    Y . Gong, A. H. Liu, H. Luo, L. Karlinsky, and J. Glass, “Joint audio and speech understanding,” in 2023 IEEE Automatic Speech Recognition and Under- standing Workshop (ASRU). IEEE, 2023, pp. 1–8

    2023

  17. [25]

    Robust speech recognition via large- scale weak supervision (arxiv: 2212.04356). arxiv,

    A. Radford, J. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large- scale weak supervision (arxiv: 2212.04356). arxiv,” 2022

    2022 arXiv

  18. [26]

    Llark: A multimodal instruction-following language model for music,

    J. P. Gardner, S. Durand, D. Stoller, and R. M. Bittner, “Llark: A multimodal instruction-following language model for music,” in International Conference on Ma- chine Learning. PMLR, 2024, pp. 15 037–15 082

    2024

  19. [27]

    Pengi: An audio language model for audio tasks,

    S. Deshmukh, B. Elizalde, R. Singh, and H. Wang, “Pengi: An audio language model for audio tasks,” Advances in Neural Information Processing Systems , vol. 36, pp. 18 090–18 108, 2023

    2023

  20. [28]

    Gama: A large audio-language model with advanced audio understanding and complex reasoning abilities,

    S. Ghosh, S. Kumar, A. Seth, C. K. R. Evuru, U. Tyagi, S. Singh, O. Nieto, R. Duraiswami, and D. Manocha, “Gama: A large audio-language model with advanced audio understanding and complex reasoning abilities,” CoRR, 2024

    2024

  21. [29]

    Qwen2-audio technical re- port,

    Y . Chu, J. Xu, Q. Yang, H. Wei, X. Wei, Z. Guo, Y . Leng, Y . Lv, J. He, J. Linet al., “Qwen2-audio technical re- port,” arXiv preprint arXiv:2407.10759, 2024

    2024 arXiv

  22. [30]

    Audio flamingo: a novel audio language model with few-shot learning and dialogue abilities,

    Z. Kong, A. Goel, R. Badlani, W. Ping, R. Valle, and B. Catanzaro, “Audio flamingo: a novel audio language model with few-shot learning and dialogue abilities,” in Proceedings of the 41st International Conference on Machine Learning, 2024, pp. 25 125–25 148

    2024

  23. [31]

    Audio flamingo 2: An audio-language model with long-audio understanding and expert reasoning abili- ties,

    S. Ghosh, Z. Kong, S. Kumar, S. Sakshi, J. Kim, W. Ping, R. Valle, D. Manocha, and B. Catanzaro, “Audio flamingo 2: An audio-language model with long-audio understanding and expert reasoning abili- ties,” arXiv preprint arXiv:2503.03983, 2025

    2025 arXiv

  24. [32]

    Mir_eval: A transparent implementation of common mir metrics

    C. Raffel, B. McFee, E. J. Humphrey, J. Salamon, O. Ni- eto, D. Liang, D. P. Ellis, and C. C. Raffel, “Mir_eval: A transparent implementation of common mir metrics.” in ISMIR, 2014, pp. 367–372

    2014

  25. [33]

    Two data sets for tempo estimation and key detection in electronic dance music annotated from user corrections,

    P. Knees, Á. Faraldo Pérez, H. Boyer, R. V ogl, S. Böck, F. Hörschläger, M. Le Goff et al., “Two data sets for tempo estimation and key detection in electronic dance music annotated from user corrections,” in Proceedings of the 16th International Society for Music Informa- tio...

    2015

  26. [34]

    1000 songs for emotional analysis of music,

    M. Soleymani, M. N. Caro, E. M. Schmidt, C.-Y . Sha, and Y .-H. Yang, “1000 songs for emotional analysis of music,” in Proceedings of the 2nd ACM international workshop on Crowdsourcing for multimedia, 2013, pp. 1–6

    2013

  27. [35]

    Evaluation of algorithms using games: The case of music tagging

    E. Law, K. West, M. I. Mandel, M. Bay, and J. S. Downie, “Evaluation of algorithms using games: The case of music tagging.” in ISMIR. Citeseer, 2009, pp. 387–392

    2009

  28. [36]

    The mtg-jamendo dataset for automatic mu- sic tagging,

    D. Bogdanov, M. Won, P. Tovstogan, A. Porter, and X. Serra, “The mtg-jamendo dataset for automatic mu- sic tagging,” in International Conference on Machine Learning. ICML, 2019

    2019

  29. [37]

    Neural au- dio synthesis of musical notes with wavenet autoen- coders,

    J. Engel, C. Resnick, A. Roberts, S. Dieleman, M. Norouzi, D. Eck, and K. Simonyan, “Neural au- dio synthesis of musical notes with wavenet autoen- coders,” in International Conference on Machine Learn- ing. PMLR, 2017, pp. 1068–1077

    2017

  30. [38]

    Musical genre classifica- tion of audio signals,

    G. Tzanetakis and P. Cook, “Musical genre classifica- tion of audio signals,”IEEE Transactions on speech and audio processing, vol. 10, no. 5, pp. 293–302, 2002

    2002

  31. [39]

    V ocalset: A singing voice dataset

    J. Wilkins, P. Seetharaman, A. Wahl, and B. Pardo, “V ocalset: A singing voice dataset.” inISMIR, 2018, pp. 468–474

    2018

  32. [40]

    The song describer dataset: a corpus of au- dio captions for music-and-language evaluation,

    I. Manco, B. Weck, S. Doh, M. Won, Y . Zhang, D. Bog- danov, Y . Wu, K. Chen, P. Tovstogan, E. Benetos et al. , “The song describer dataset: a corpus of au- dio captions for music-and-language evaluation,” arXiv preprint arXiv:2311.10057, 2023

    2023 arXiv

  33. [41]

    Musiclm: Generat- ing music from text,

    A. Agostinelli, T. I. Denk, Z. Borsos, J. Engel, M. Verzetti, A. Caillon, Q. Huang, A. Jansen, A. Roberts, M. Tagliasacchi et al., “Musiclm: Generat- ing music from text,” arXiv preprint arXiv:2301.11325, 2023

    2023 arXiv

  34. [42]

    Automatic lyric tran- scription from karaoke vocal tracks: Resources and a baseline system,

    G. Roa Dabike and J. Barker, “Automatic lyric tran- scription from karaoke vocal tracks: Resources and a baseline system,” in Proceedings of the 20th Annual Conference of the International Speech Communication Association (INTERSPEECH 2019), 2019

    2019

  35. [43]

    An experimental com- parison of audio tempo induction algorithms,

    F. Gouyon, A. Klapuri, S. Dixon, M. Alonso, G. Tzane- takis, C. Uhle, and P. Cano, “An experimental com- parison of audio tempo induction algorithms,” IEEE Transactions on Audio, Speech, and Language Process- ing, vol. 14, no. 5, pp. 1832–1844, 2006

    2006

  36. [44]

    Rhythmic pattern modeling for beat and downbeat tracking in musical audio

    F. Krebs, S. Böck, and G. Widmer, “Rhythmic pattern modeling for beat and downbeat tracking in musical audio.” in Ismir, 2013, pp. 227–232

    2013

  37. [45]

    Medleydb: A multitrack dataset for annotation-intensive mir research,

    R. Bittner, J. Salamon, M. Tierney, M. Mauch, C. Can- nam, and J. Bello, “Medleydb: A multitrack dataset for annotation-intensive mir research,” 10 2014

    2014

  38. [46]

    Frame-level multi-label playing technique detection us- ing multi-scale network and self-attention mechanism,

    D. Li, M. Che, W. Meng, Y . Wu, Y . Yu, F. Xia, and W. Li, “Frame-level multi-label playing technique detection us- ing multi-scale network and self-attention mechanism,” in IEEE International Conference on Acoustics, Speech and Signal Processing ICASSP 2023, Rhodes Island, Gr...

    2023

  39. [47]

    C-pack: Packed resources for general chinese embeddings,

    S. Xiao, Z. Liu, P. Zhang, N. Muennighoff, D. Lian, and J.-Y . Nie, “C-pack: Packed resources for general chinese embeddings,” in Proceedings of the 47th in- ternational ACM SIGIR conference on research and development in information retrieval, 2024, pp. 641– 649

    2024

  40. [48]

    Bleu: a method for automatic evaluation of machine translation,

    K. Papineni, S. Roukos, T. Ward, and W. jing Zhu, “Bleu: a method for automatic evaluation of machine translation,” in Proceedings of the 40th annual meeting of the Association for Computational Linguistics, 2002, pp. 311–318

    2002

  41. [49]

    ORANGE: a method for evaluating automatic evaluation metrics for machine translation,

    C.-Y . Lin and F. J. Och, “ORANGE: a method for evaluating automatic evaluation metrics for machine translation,” in COLING 2004: Proceedings of the 20th International Conference on Computational Linguistics. Geneva, Switzerland: COLING, aug 23–aug 27 2004, pp. 501–507. [Onlin...

    2004

  42. [50]

    Meteor: An automatic metric for mt evaluation with improved correlation with human judgments,

    S. Banerjee and A. Lavie, “Meteor: An automatic metric for mt evaluation with improved correlation with human judgments,” in Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, 2005, pp. 65–72

    2005

  43. [51]

    ROUGE: A package for automatic evaluation of summaries,

    C.-Y . Lin, “ROUGE: A package for automatic evaluation of summaries,” in Text Summarization Branches Out . Barcelona, Spain: Association for Computational Linguistics, Jul. 2004, pp. 74–81. [Online]. Available: https://www.aclweb.org/anthology/ W04-1013

    2004

  44. [52]

    Bertscore: Evaluating text generation with bert,

    T. Zhang*, V . Kishore*, F. Wu*, K. Q. Weinberger, and Y . Artzi, “Bertscore: Evaluating text generation with bert,” in International Conference on Learning Representations, 2020. [Online]. Available: https: //openreview.net/forum?id=SkeHuCVFDr

    2020

  45. [53]

    End-to-end musical key estimation using a convolutional neural network,

    F. Korzeniowski and G. Widmer, “End-to-end musical key estimation using a convolutional neural network,” in 2017 25th European Signal Processing Conference (EUSIPCO). IEEE, 2017, pp. 966–970

    2017

  46. [54]

    Codified audio language modeling learns useful representa- tions for music information retrieval,

    R. Castellon, C. Donahue, and P. Liang, “Codified audio language modeling learns useful representa- tions for music information retrieval,” arXiv preprint arXiv:2107.05677, 2021

    2021 arXiv

  47. [55]

    Marble: Music audio representation benchmark for universal evaluation,

    R. Yuan, Y . Ma, Y . Li, G. Zhang, X. Chen, H. Yin, Y . Liu, J. Huang, Z. Tian, B. Deng et al. , “Marble: Music audio representation benchmark for universal evaluation,”Advances in Neural Information Processing Systems, vol. 36, pp. 39 626–39 647, 2023

    2023

  48. [56]

    Mulan: A joint embedding of music audio and natural language,

    Q. Huang, A. Jansen, J. Lee, R. Ganti, J. Y . Li, and D. P. Ellis, “Mulan: A joint embedding of music audio and natural language,” arXiv preprint arXiv:2208.12415 , 2022

    2022 arXiv

  49. [57]

    Supervised and un- supervised learning of audio representations for music understanding,

    M. C. McCallum, F. Korzeniowski, S. Oramas, F. Gouyon, and A. F. Ehmann, “Supervised and un- supervised learning of audio representations for music understanding,” Ismir 2022 Hybrid Conference, 2022

    2022

  50. [58]

    Music representation learning based on editorial metadata from discogs,

    P. Alonso-Jiménez, X. Serra, and D. Bogdanov, “Music representation learning based on editorial metadata from discogs,” 2022

    2022

  51. [59]

    Masked modeling duo: Learning repre- sentations by encouraging both networks to model the input,

    D. Niizumi, D. Takeuchi, Y . Ohishi, N. Harada, and K. Kashino, “Masked modeling duo: Learning repre- sentations by encouraging both networks to model the input,” in ICASSP 2023-2023 IEEE International Con- ference On Acoustics, Speech And Signal Processing (ICASSP). IEEE, 202...

    2023

  52. [60]

    Mert: Acoustic music understanding model with large-scale self-supervised training,

    Y . Li, R. Yuan, G. Zhang, Y . Ma, X. Chen, H. Yin, C. Lin, A. Ragni, E. Benetos, N. Gyenge, R. Dannen- berg, R. Liu, W. Chen, G. Xia, Y . Shi, W. Huang, Y . Guo, and J. Fu, “Mert: Acoustic music understanding model with large-scale self-supervised training,” 2023

    2023

  53. [61]

    Towards learning universal audio repre- sentations,

    L. Wang, P. Luc, Y . Wu, A. Recasens, L. Smaira, A. Brock, A. Jaegle, J.-B. Alayrac, S. Dieleman, J. Car- reira et al., “Towards learning universal audio repre- sentations,” in ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). ...

    2022

  54. [62]

    Lp-musiccaps: Llm-based pseudo music captioning,

    S. Doh, K. Choi, J. Lee, and J. Nam, “Lp-musiccaps: Llm-based pseudo music captioning,” in Ismir 2023 Hybrid Conference, 2023

    2023

  55. [63]

    Transfer learning of wav2vec 2.0 for automatic lyric transcription,

    L. Ou, X. Gu, and Y . Wang, “Transfer learning of wav2vec 2.0 for automatic lyric transcription,” in Is- mir 2022 Hybrid Conference, 2022

    2022

  56. [64]

    Beatnet: CRNN and particle filtering for online joint beat, downbeat and meter tracking,

    M. Heydari, F. Cwitkowitz, and Z. Duan, “Beatnet: CRNN and particle filtering for online joint beat, downbeat and meter tracking,” in Proceedings of the 22nd International Society for Music Information Retrieval Conference, ISMIR 2021, Online, November 7-12, 2021 , J. H. Lee, ...

    2021

  57. [65]

    Beat transformer: Demixed beat and downbeat tracking with dilated self-attention,

    J. Zhao, G. Xia, and Y . Wang, “Beat transformer: Demixed beat and downbeat tracking with dilated self-attention,” in Proceedings of the 23rd International Society for Music Information Retrieval Conference, ISMIR 2022, Bengaluru, India, December 4-8, 2022, P. Rao, H. A. Murth...

    2022

  58. [66]

    Hkdsme: Het- erogeneous knowledge distillation for semi-supervised singing melody extraction using harmonic supervision,

    S. Yu, X. He, K. Chen, and Y . Yu, “Hkdsme: Het- erogeneous knowledge distillation for semi-supervised singing melody extraction using harmonic supervision,” in ACM Multimedia 2024

    2024

  59. [67]

    Mertech: Instrument playing technique detection using self-supervised pretrained model with multi-task finetuning,

    D. Li, Y . Ma, W. Wei, Q. Kong, Y . Wu, M. Che, F. Xia, E. Benetos, and W. Li, “Mertech: Instrument playing technique detection using self-supervised pretrained model with multi-task finetuning,” in ICASSP, 2024

    2024

  60. [68]

    Lyricwhiz: Ro- bust multilingual lyrics transcription by whispering to chatgpt

    L. Zhuo, R. Yuan, J. Pan, Y . Ma, Y . Li, G. Zhang, S. Liu, R. Dannenberg, J. Fu, C. Lin et al., “Lyricwhiz: Ro- bust multilingual lyrics transcription by whispering to chatgpt.” International Society for Music Information Retrieval Conference (ISMIR), 2023. E. SUPPLEMENTARY M...

    2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.