Pith. sign in

REVIEW 9 cited by

ChatMusician: Understanding and Generating Music Intrinsically with LLM

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2402.16153 v1 pith:F2CWKXNQ submitted 2024-02-25 cs.SD cs.AIcs.CLcs.LGcs.MMeess.AS

Ruibin Yuan , Hanfeng Lin , Yi Wang , Zeyue Tian , Shangda Wu , Tianhao Shen , Ge Zhang , Yuhang Wu
show 27 more authors
Cong Liu Ziya Zhou Ziyang Ma Liumeng Xue Ziyu Wang Qin Liu Tianyu Zheng Yizhi Li Yinghao Ma Yiming Liang Xiaowei Chi Ruibo Liu Zili Wang Pengfei Li Jingcheng Wu Chenghua Lin Qifeng Liu Tao Jiang Wenhao Huang Wenhu Chen Emmanouil Benetos Jie Fu Gus Xia Roger Dannenberg Wei Xue Shiyin Kang Yike Guo
This is my paper · ORCID
classification cs.SDcs.AIcs.CLcs.LGcs.MMeess.AS
keywords musicchatmusicianlanguageabilitiesmusicalllama2llmsmodel
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

While Large Language Models (LLMs) demonstrate impressive capabilities in text generation, we find that their ability has yet to be generalized to music, humanity's creative language. We introduce ChatMusician, an open-source LLM that integrates intrinsic musical abilities. It is based on continual pre-training and finetuning LLaMA2 on a text-compatible music representation, ABC notation, and the music is treated as a second language. ChatMusician can understand and generate music with a pure text tokenizer without any external multi-modal neural structures or tokenizers. Interestingly, endowing musical abilities does not harm language abilities, even achieving a slightly higher MMLU score. Our model is capable of composing well-structured, full-length music, conditioned on texts, chords, melodies, motifs, musical forms, etc, surpassing GPT-4 baseline. On our meticulously curated college-level music understanding benchmark, MusicTheoryBench, ChatMusician surpasses LLaMA2 and GPT-3.5 on zero-shot setting by a noticeable margin. Our work reveals that LLMs can be an excellent compressor for music, but there remains significant territory to be conquered. We release our 4B token music-language corpora MusicPile, the collected MusicTheoryBench, code, model and demo in GitHub.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 9 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Music I Care About: Automated Multimodal Benchmarking of LLM Music Perception Skills on (Almost) Any Music

    cs.SD 2026-07 unverdicted novelty 6.0 of 10

    A meta-benchmark that auto-generates multimodal music-perception multiple-choice tests from user symbolic music, demonstrated on ChoraleBricks with text-only and white-noise controls.

  2. MusiXQA: Advancing Visual Music Understanding in Multimodal Large Language Models

    cs.CV 2025-06 conditional novelty 6.0 of 10

    A synthetic music sheet QA dataset and a LoRA-fine-tuned Phi-3 model show large accuracy gains on OMR and chord tasks, but only within the synthetic distribution.

  3. Can Large Language Models Predict Audio Effects Parameters from Natural Language?

    cs.SD 2025-05 conditional novelty 6.0 of 10

    LLMs can predict equalizer and reverb parameters from natural language descriptions, and adding DSP features, DSP function code, and few-shot examples improves the predictions.

  4. Via Score to Performance: Efficient Human-Controllable Long Song Generation with Bar-Level Symbolic Notation

    cs.SD 2025-08 unverdicted novelty 5.0 of 10

    A bar-level symbolic-score song generator (BACH) is claimed to beat published systems and commercial Suno on human-rated quality, duration, and efficiency, but the supporting full text is corrupted and unverifiable.

  5. Exploring GPT's Ability as a Judge in Music Understanding

    cs.IR 2025-01 conditional novelty 5.0 of 10

    GPT-3.5 detects synthetic annotation errors in beat, chord, and key tasks above random chance, with accuracy partly influenced by musical concepts in the prompt.

  6. WeaveMuse: An Open Agentic System for Multimodal Music Understanding and Generation

    cs.SD 2025-09 reject novelty 4.0 of 10

    An open multi-agent system that orchestrates specialized music models for understanding, composition, and synthesis, with local or hosted deployment.

  7. CoComposer: LLM Multi-agent Collaborative Music Composition

    cs.SD 2025-08 conditional novelty 4.0 of 10

    A five-agent LLM system for ABC-notation composition scores modestly higher than ComposerX and a single LLM on an automated aesthetic model, but no error bars or significance tests are reported.

  8. Are Large Language Models Capable of Deep Relational Reasoning? Insights from DeepSeek-R1 and Benchmark Comparisons

    cs.AI 2025-06 conditional novelty 3.0 of 10

    DeepSeek-R1 outperforms GPT-4o and DeepSeek-V3 on family tree and graph reasoning benchmarks at sizes 10 and 20, but all models collapse at size 40.

  9. Content filtering methods for music recommendation: A review

    cs.IR 2025-07 conditional

    A survey of content-based music recommendation methods, including audio analysis, lyrics analysis, and context awareness, with no new experimental results.

Pith tools