REVIEW 10 cited by
MuMu-LLaMA: Multi-modal Music Understanding and Generation via Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Research on large language models has advanced significantly across text, speech, images, and videos. However, multi-modal music understanding and generation remain underexplored due to the lack of well-annotated datasets. To address this, we introduce a dataset with 167.69 hours of multi-modal data, including text, images, videos, and music annotations. Based on this dataset, we propose MuMu-LLaMA, a model that leverages pre-trained encoders for music, images, and videos. For music generation, we integrate AudioLDM 2 and MusicGen. Our evaluation across four tasks--music understanding, text-to-music generation, prompt-based music editing, and multi-modal music generation--demonstrates that MuMu-LLaMA outperforms state-of-the-art models, showing its potential for multi-modal music applications.
Forward citations
Cited by 10 Pith papers
-
MusTBENCH: Benchmarking and Advancing Temporal Grounding in Music LLMs
MusTBENCH evaluates temporal grounding in large audio-language models via five expert-validated tasks, and MusT improves performance through encoder adaptation, LLM adaptation, supervised fine-tuning, and RL optimization.
-
TORUS: A Test of Rendering-Understanding Self-Coherence for Unified Audio Models
A new audio benchmark, TORUS, shows unified audio models are not self-coherent: the best model answers 50.5% of questions about its own generations, below a 63.2% cascaded specialist baseline.
-
JenBridge: Adaptive Long-Form Video Soundtracking across Scene Transitions
JenBridge pretrains a flow-matching Transformer on text-audio data then adapts it with video conditioning and an LLM director to select transitions, claiming better coherence than prior methods on a new LVS benchmark.
-
Assessing Factual Music Comprehension in Large Audio Language Models
Standard NLP metrics fail to capture factual music understanding in audio-language models; a CLAP-based metric and an LLM-parsed factual QA protocol measure it more directly.
-
Zero-Effort Image-to-Music Generation: An Interpretable RAG-based VLM Approach
A zero-training VLM framework generates music from images via ABC notation, multi-modal RAG, and self-refinement while providing text and visual explanations for the outputs.
-
AudioX-Turbo: A Unified Framework for Efficient Anything-to-Audio Generation
AudioX-Turbo distills a Multimodal Diffusion Transformer into a 4-step student model for efficient multimodal anything-to-audio generation, trained on a new 9.2M-sample dataset IF-caps-Pro.
-
AudioX-Turbo: A Unified Framework for Efficient Anything-to-Audio Generation
A distilled multimodal diffusion model generates audio from text, video, or audio in four steps with claimed superior quality and ~25× fewer function evaluations.
-
EntangleCodec: A Unified Discrete Audio Tokenizer via Semantic-Acoustic Entanglement
EntangleCodec unifies semantic and acoustic audio tokenization via caption alignment and flow-matching decoding, reporting competitive reconstruction, +7.4% gains on MMAR understanding, and 0.6B-parameter ALMs surpass...
-
WeaveMuse: An Open Agentic System for Multimodal Music Understanding and Generation
An open multi-agent system that orchestrates specialized music models for understanding, composition, and synthesis, with local or hosted deployment.
-
CoComposer: LLM Multi-agent Collaborative Music Composition
A five-agent LLM system for ABC-notation composition scores modestly higher than ComposerX and a single LLM on an automated aesthetic model, but no error bars or significance tests are reported.
Discussion (0). Sign in to comment.