REVIEW 11 cited by
M$^{2}$UGen: Multi-modal Music Understanding and Generation with the Power of Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
abstract
The current landscape of research leveraging large language models (LLMs) is experiencing a surge. Many works harness the powerful reasoning capabilities of these models to comprehend various modalities, such as text, speech, images, videos, etc. They also utilize LLMs to understand human intention and generate desired outputs like images, videos, and music. However, research that combines both understanding and generation using LLMs is still limited and in its nascent stage. To address this gap, we introduce a Multi-modal Music Understanding and Generation (M$^{2}$UGen) framework that integrates LLM's abilities to comprehend and generate music for different modalities. The M$^{2}$UGen framework is purpose-built to unlock creative potential from diverse sources of inspiration, encompassing music, image, and video through the use of pretrained MERT, ViT, and ViViT models, respectively. To enable music generation, we explore the use of AudioLDM 2 and MusicGen. Bridging multi-modal understanding and music generation is accomplished through the integration of the LLaMA 2 model. Furthermore, we make use of the MU-LLaMA model to generate extensive datasets that support text/image/video-to-music generation, facilitating the training of our M$^{2}$UGen framework. We conduct a thorough evaluation of our proposed framework. The experimental results demonstrate that our model achieves or surpasses the performance of the current state-of-the-art models.
Forward citations
Cited by 11 Pith papers
-
Exploring the Collaborative Co-Creation Process with AI: A Case Study in Novice Music Production
A qualitative case study of nine novice music producers shows AI compresses the preparation stage, adds a new 'collaging and refinement' stage, and shifts group social dynamics, leading to two proposed models.
-
Dialogue-Aware Video-to-Music Generation Using Public Domain Film Collections
A new public-domain film dataset and a frame-by-frame dialogue-conditioning module improve video-to-music generation on paired-fidelity metrics.
-
Controllable Video-to-Music Generation with Multiple Time-Varying Conditions
A two-stage video-to-music model with four time-varying controls (rhythm, melody, intensity, emotion) claims better controllability and alignment than prior V2M systems.
-
Let Your Video Listen to Your Music!
MVAA aligns a video's motion peaks to music beats via keyframe re-timing and diffusion-based inpainting, aiming to preserve the original content while improving rhythmic synchronization.
-
GVMGen: A General Video-to-Music Generation Model with Hierarchical Attentions
GVMGen generates background music from video using spatial and temporal cross-attention to condition a MusicGen decoder, reporting state-of-the-art correspondence and diversity.
-
MusiChat: Vibe Composing for Music Creation
MusiChat enables iterative, structure-preserving music editing through natural-language conversation by layering an LLM-based interface over a deterministic symbolic music engine.
-
Insect-Foundation: A Foundation Model and Large Multimodal Dataset for Vision-Language Insect Understanding
An insect-specific vision-language assistant trained on a new 1M-image multimodal insect dataset with patch-matching self-supervision reports improved insect classification and VQA accuracy over LLaVA and prior self-s...
-
VidMusician: Video-to-Music Generation with Semantic-Rhythmic Alignment via Hierarchical Visual Features
An adaptation of MusicGen that adds semantic conditioning from CLIP global features and rhythmic conditioning from CLIP local inter-frame similarity, trained in two stages, generates video-aligned background music.
-
FlowSonic: Stable Zero-Shot Music Editing via High-Order Trajectory Integration
FlowSonic combines deterministic rectified-flow inversion, cached cross-attention injection, and a 'seeded' third-order Adams-Bashforth solver to report better timbre and genre edits on small datasets.
-
Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Survey
A comprehensive survey that frames multimodal understanding and generation as next token prediction and proposes a five-part taxonomy.
-
Improving Controllability and Editability for Pretrained Text-to-Music Generation Models
A thesis compilation presenting three complementary approaches to improving editing and control of pretrained text-to-music models, with Instruct-MusicGen demonstrating the strongest stem-level editing results.
Discussion (0). Continue with ORCID to comment.