Pith. sign in

REVIEW 4 major objections 6 minor 8 cited by

CLaMP 3: Universal Music Information Retrieval Across Unaligned Modalities and Unseen Languages

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read One shared embedding space unifies sheet music, MIDI, and audio retrieval

desk verdict Solid systems paper with large released resources, but its English retrieval SOTA may be inflated by train/benchmark overlap the authors never check. read the letter →

arxiv 2502.10362 v3 pith:ZGGYMS3G submitted 2025-02-14 cs.SD eess.AS

Shangda Wu , Zhancheng Guo , Ruibin Yuan , Junyan Jiang , Seungheon Doh , Gus Xia , Juhan Nam , Xiaobing Li
show 2 more authors
Feng Yu Maosong Sun
This is my paper · ORCID
classification cs.SDeess.AS
keywords musicinformationretrievalcontrastivelearningcross-modalmultilingualsheetaudiorepresentationretrieval-augmentedgeneration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

CLaMP 3 claims that a single contrastively trained model can place sheet music, performance MIDI, audio recordings, and text in one embedding space, so a text query retrieves any music modality and unpaired modalities can retrieve each other through text as a bridge. It also claims strong cross-lingual retrieval in ten languages, five of which are never aligned with music during training. To reach this, the paper builds M4-RAG, a 2.31-million-pair music-text dataset produced by retrieval-augmented LLM annotation, and WikiMT-X, a new benchmark of 1,000 sheet-audio-text triplets with four semantic text types. If the claims hold, text-based search across heterogeneous music formats and low-resource languages becomes feasible without collecting paired data for every modality-language combination.

What carries the argument

The load-bearing mechanism is a multi-stage InfoNCE contrastive training schedule that uses text as a stable anchor: the multilingual text encoder (XLM-R-base) is first aligned with the symbolic music encoder, then frozen while the audio encoder is attached, then unfrozen for refinement, and finally frozen again to repair alignment drift in the first modality. Freezing the text encoder between stages prevents the two music modalities from pulling it in conflicting directions, and the shared text space then serves as the bridge through which unpaired modalities such as audio and sheet music can retrieve each other. Supporting this is the M4-RAG pipeline, which retrieves web documents from titles and artists, generates structured multilingual annotations with an LLM, filters and translates them, and yields 2.31 million music-text pairs across 27 languages and 194 countries.

What would settle it

Retrieve with native human-written queries in a language entirely absent from XLM-R's pretraining corpus (for example Tigrinya or Wolof) and compare MRR against English and against the reported translated-language scores; if performance collapses toward random or tracks translation BLEU rather than language structure, the unseen-language generalization claim fails. A cheaper check is to recompute Table 3 after removing from each translated query any tokens that also appear in the English annotation, to see how much of the measured retrieval is surface-form leakage.

Watch

Extended reading notes

Core claim

The central claim is that contrastive learning with text as a stable anchor produces a universal music embedding space: the same multilingual text encoder aligns both a symbolic music encoder (ABC sheet music and performance MIDI) and an audio encoder, and the alignment transfers to modality pairs that were never directly supervised. Concretely, CLaMP 3c2sa raises WikiMT text-to-sheet MRR from 0.3438 (CLaMP 2) to 0.4498, and CLaMP 3saas raises Song Describer Dataset text-to-audio MRR from 0.1437 (TTMR++) to 0.1985. On multilingual retrieval, the audio-oriented variant outperforms an indirectly pretrained variant on Finnish, Greek, and Kazakh, languages marked unseen during alignment, and its weakest unseen language, Amharic, still beats CLAP's English-only audio retrieval. Cross-modal directions such as sheet-to-audio and MIDI-to-audio, never explicitly trained, reach MRR values well above the 0.0075 random baseline, which the paper interprets as evidence that text stitches all modalities into a single shared space.

Load-bearing premise

The claim of generalizing to unseen languages rests on treating five asterisked languages as genuinely unexposed during alignment even though the text encoder was pretrained on text in those languages, and their evaluation data is machine-translated and noisy, so the high scores could reflect translation artifacts and surface-form overlap with English rather than true zero-shot language transfer.

Editorial extensions

If this is right

  • Text-based search can retrieve sheet music, MIDI performance files, and audio recordings with one model, so a single system replaces separate text-to-audio and text-to-symbolic retrieval pipelines.
  • Because text acts as a bridge, queries need not be in the same modality as the target: a sheet music description can surface audio recordings and vice versa without any paired symbolic-audio training data.
  • Retrieval in low-resource languages becomes possible to the extent that the text encoder's pretrained multilingual semantics cover them, enabling search for music described in Finnish, Greek, Kazakh, Tamil, or Amharic.
  • The M4-RAG dataset and WikiMT-X benchmark provide a reusable 2.31-million-pair training resource and a first trilingual-modal evaluation standard for future music retrieval and captioning work.
  • LLM-generated, retrieval-augmented annotations can substitute for expensive human-written music captions at scale, as evidenced by the model's gains on benchmarks built from both human and generated text.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The 'unseen language' results are better read as cross-lingual transfer inherited from XLM-R's pretraining plus robustness to machine-translation noise, rather than zero-shot generalization to genuinely new languages; the five asterisked languages are absent from music-text alignment but present in the encoder's pretraining corpus, and their evaluation text is machine-translated with back-translat
  • The absolute MRR values for symbolic-to-audio and audio-to-symbolic retrieval (roughly 0.03 to 0.07) sit far above random but are too low for practical search, and the paper itself concedes this gap in its limitations; closing it likely requires supervised symbolic-audio pairing rather than text bridging alone.
  • Because M4-RAG annotations are generated from web search results by an LLM, the dataset inherits whatever genre and regional biases exist in online music documentation, so the 194-country coverage reflects search-document availability more than musical diversity, which subsequent work on the dataset should verify.
  • A testable extension: replace the machine-translated multilingual queries with native human-written annotations in the same languages and re-measure MRR; if performance tracks translation BLEU rather than language relatedness, the cross-lingual generalization claim will need to be restated as a translation-robustness result.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. CLaMP 3 is a unified music information retrieval model that uses contrastive learning (InfoNCE) to align sheet music (ABC notation), performance signals (MIDI), audio, and multilingual text in a shared embedding space. The text encoder is XLM-R-base; symbolic and audio encoders are transformer-based, with frozen MERT features for audio. A multi-stage training recipe alternates freezing and unfreezing the text encoder to mitigate modality interference. To obtain training data, the authors curate M4-RAG, a set of 2.31 million music-text entries generated by retrieval-augmented LLM annotation from titles, artists, and web search results, with multilingual translations. They also release WikiMT-X, a benchmark of 1,000 triplets of ABC, MIDI, audio, and four types of text annotations. Experiments report state-of-the-art English text-to-music retrieval on WikiMT, MidiCaps, SDD, and MusicCaps-Remake; multilingual retrieval in ten languages, including five not used in M4-RAG alignment; emergent cross-modal retrieval; and linear-probe classification. The paper includes a detailed MusicCaps leakage analysis and human evaluation of WikiMT-X annotation quality.

Significance. The paper makes a useful advance: it is, to my knowledge, the first system to demonstrate retrieval across symbolic, performance, and audio modalities with a single multilingual text bridge, and it releases substantial artifacts (code, weights, 1.56 million audio-text pairs, and the WikiMT-X benchmark). The multi-stage alignment ablation and the leakage-aware MusicCaps evaluation are thoughtful, and the authors deserve credit for being transparent about the weak audio-symbolic alignment and about translation dependence in the Limitations section. However, the headline empirical claims depend on two unresolved issues: possible train/test overlap between M4-RAG and WikiMT/WikiMT-X, and the interpretation of 'unseen languages' in a pipeline whose text encoder has already been pretrained on those languages and whose evaluation uses machine translation. These issues are fixable, so the contribution is potentially strong, but the current evidence does not yet fully support the 'universal' and 'state-of-the-art' claims as stated.

major comments (4)
  1. [Section 3.1, Section 3.4, Tables 2 and 3] The manuscript does not document any exclusion of WikiMT or WikiMT-X items from M4-RAG, even though M4-RAG's symbolic component is sourced from WebMT (Section 3.1) and WikiMT-X is described as extending WikiMT (Section 3.4). Since WikiMT is a curated subset of WebMT, the 1,000 WikiMT-X ABC lead sheets (and likely their matched audio) may appear verbatim in CLaMP 3's training set. Appendix F carefully addresses leakage for MusicCaps and SDD but is silent on WikiMT-X. If these items are in the training set, the WikiMT and WikiMT-X retrieval MRRs in Tables 2 and 3 could reflect memorization rather than generalization, undermining the central state-of-the-art claim. The authors should provide an exact-match and near-duplicate overlap analysis for ABC, MIDI, and audio, and ideally retrain or fine-tune after removing overlapping items.
  2. [Section 4.3, Table 3] The term 'unseen languages' is used for fi, el, ta, kk, and am, but XLM-R-base is pretrained on CommonCrawl data covering these languages (Section 2.2), and CLaMP 3c2sa is initialized from CLaMP 2, which aligned symbolic music with all XLM-R-supported languages. Consequently, only CLaMP 3saas is unexposed at the music-text alignment level, and even that model's text encoder has seen these languages. Moreover, all multilingual test text is generated by SeamlessM4T translation of English WikiMT-X annotations, with back-translation BLEU as low as 36.57; this introduces translation noise and leaves surface-form overlap (genre terms, numbers, proper names) with English training annotations. The cross-lingual generalization claim should be reworded as 'languages unseen in music-text contrastive training,' and the evaluation should include native-speaker verification or a control using native-language annotations.
  3. [Section 4.1, Appendix D] Section 4.1 reports the two 'top-performing variants' among six training orders, but the selection criterion is not specified and no validation set is described. If the two variants were chosen based on their scores on the same WikiMT, MidiCaps, SDD, and MC-R test sets used for the headline comparisons, the reported state-of-the-art numbers are optimistically biased. The authors should state the selection procedure, use a held-out validation split for choosing variants and hyperparameters, or report results for all six variants on the primary comparisons with appropriate multiple-comparison awareness.
  4. [Tables 2 and 3, Appendix D] All retrieval comparisons are single point estimates without confidence intervals or significance tests, and the choice of reported variant is not consistently the best among the authors' own models: on SDD audio retrieval, CLaMP 3assa achieves 0.2003 while the reported audio-optimized CLaMP 3saas achieves 0.1985 (Table 6 in Appendix D). Repeating retrieval evaluations across random seeds and reporting paired significance tests would substantially strengthen the claim that CLaMP 3 'significantly surpasses' baselines.
minor comments (6)
  1. [Table 2 caption] The caption states that WikiMT and MidiCaps have 1,010 pairs, while WikiMT-X is described as having 1,000 entries; please clarify the relationship between these counts and the 10 duplicates removed during WikiMT-X construction.
  2. [Notation throughout] The notation 'CLaMP 3 c2sa' appears with inconsistent spacing and superscript formatting across the text and tables; please standardize it.
  3. [Section 4.2] The phrase 'despite using only half the training data' lacks an explicit comparison; state the training-set sizes for CLaMP 2 and CLaMP 3 so the reader can verify the claim.
  4. [Figures 3 and 4] The log-scale count axes in Figures 3 and 4 have no visible tick labels; adding labeled axes or a caption note would improve readability.
  5. [Appendix F] The statement that only 150 M4-RAG audio recordings overlap with MusicCaps is not accompanied by a description of how those overlaps were identified; a brief description would aid reproducibility.
  6. [Table 5] The layout of Table 5 makes it difficult to determine which Aesthetics value belongs to which dataset and annotation type; please reformat the table.

Circularity Check

1 steps flagged · score 4.0 of 10

No mathematical derivation is circular, but the WikiMT/WikiMT-X evaluation is built from the same WebMT/audio pool used to train M4-RAG, so the headline retrieval numbers are partly self-referential.

  1. fitted input called prediction [Sections 3.1, 3.4, and 4.1; Tables 2 and 3]
    "The symbolic music data is sourced from WebMusicText (WebMT) (Wu et al., 2023a) with 1.4 million ABC notation files ... WikiMT-X (WikiMusicText-eXtended) extends WikiMT (Wu et al., 2023a), focusing on 20th-century Western music with 1,000 entries, each with sheet music, audio, and detailed metadata. ... M4-RAG was divided into 99% for training and 1% for validation."

    WikiMT and WikiMT-X are not established as held-out with respect to M4-RAG: the symbolic side of M4-RAG is WebMT, and WikiMT-X is presented as an extension of WikiMT, with both WebMT and WikiMT attributed to the same Wu et al. (2023a) reference. The paper never states that the 1,000 WikiMT-X ABC files (or their YouTube-matched audio) were excluded from the 1.4M WebMT files, the 1.5M MIDI files, or the 160k-hour audio pool. Consequently the Table 2 and Table 3 MRR gains on WikiMT and WikiMT-X could reflect retrieval of training items rather than generalization, making the 'state-of-the-art' claim on those columns dependent on the training set by construction.

full rationale

The model itself is presented as an empirical contrastive-learning system rather than a theorem-derived result, so no equation-level circularity is present. The main circularity concern is evaluative: the headline WikiMT and WikiMT-X retrieval scores are produced by a model trained on a dataset that appears to contain the same symbolic items (and potentially the same audio) as the benchmark, without any documented overlap analysis. This makes those specific numbers partially self-referential. However, the central claims also rest on external benchmarks (MidiCaps, SDD, MusicCaps-Remake) and on the released M4-RAG resource, which provide independent content; the cross-lingual 'unseen' phrasing is an overstatement about XLM-R pretraining rather than a circular derivation. Score 4 reflects one load-bearing self-referential evaluation set while acknowledging that much of the paper's empirical contribution is independent.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new theoretical entities. Its free parameters are mostly dataset-construction decisions and the un-reported contrastive temperature. The main structural assumptions are the ImageBind-style text-bridge assumption, the use of frozen MERT features, and the use of XLM-R for multilingual generalization. The absence of an explicit value for tau in Eq. 1 is a notable gap in reproducibility.

free parameters (4)
  • contrastive temperature tau = not reported
    The InfoNCE loss in Eq. 1 uses a temperature parameter tau, but no value is given anywhere in the paper; its choice affects the learned embedding geometry.
  • training order / variant selection for the two best models = symbolic -> audio -> audio -> symbolic (saas); and c2-sa
    Six variants were trained (Appendix D) and the two best-performing ones were selected and presented in the main tables. The selection is performed on the same evaluation benchmarks that are used to report SOTA; this is a mild form of model selection on the test data.
  • number of RAG search results (top 10) = 10
    Chosen by the authors during dataset construction; affects annotation quality.
  • M4-RAG quality filters = Boolean sufficient_information, non-empty fields, language consistency, XLM-R/Qwen support
    The threshold for 'sufficient_information' is decided by Qwen-2.5-72B and the authors' prompt, not objectively specified; this controls the final dataset composition.
assumptions (4)
  • domain assumption The shared embedding space learned by contrastive learning generalizes across modalities to unaligned pairs (text as a bridge).
    This is the ImageBind-style assumption underlying the whole method and the emergent cross-modal retrieval experiments; if the text-anchored space does not align modalities in unseen directions, the retrieval claims fail.
  • domain assumption XLM-R-base's cross-lingual representations make unseen languages usable without music-text alignment.
    The paper relies on XLM-R being trained on 100 languages (Section 2.2), so that unseen languages still produce useful text embeddings. This is empirically plausible but assumed.
  • domain assumption Machine translation preserves enough semantics that retrieval on translated annotations is a valid multilingual benchmark.
    Section 4.3 and Limitations use SeamlessM4T translations because no native multilingual music-text benchmarks exist; low BLEU for some languages (ta: 40.07, kk: 36.57) is acknowledged in Limitations.
  • domain assumption MERT-v1-95M frozen features are sufficient for audio representation.
    The audio encoder is trained from scratch on top of frozen MERT features rather than raw audio; the paper assumes MERT features are informative enough for text-audio retrieval.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CLaMP 3: Universal Music Information Retrieval Across Unaligned Modalities and Unseen Languages." pith.science (2026). https://pith.science/paper/ZGGYMS3G

@misc{pith2026250210362,
  author       = {Pith},
  title        = {Pith review of: CLaMP 3: Universal Music Information Retrieval Across Unaligned Modalities and Unseen Languages},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZGGYMS3G}},
  note         = {Machine review of arXiv:2502.10362}
}
read the original abstract

CLaMP 3 is a unified framework developed to address challenges of cross-modal and cross-lingual generalization in music information retrieval. Using contrastive learning, it aligns all major music modalities--including sheet music, performance signals, and audio recordings--with multilingual text in a shared representation space, enabling retrieval across unaligned modalities with text as a bridge. It features a multilingual text encoder adaptable to unseen languages, exhibiting strong cross-lingual generalization. Leveraging retrieval-augmented generation, we curated M4-RAG, a web-scale dataset consisting of 2.31 million music-text pairs. This dataset is enriched with detailed metadata that represents a wide array of global musical traditions. To advance future research, we release WikiMT-X, a benchmark comprising 1,000 triplets of sheet music, audio, and richly varied text descriptions. Experiments show that CLaMP 3 achieves state-of-the-art performance on multiple MIR tasks, significantly surpassing previous strong baselines and demonstrating excellent generalization in multimodal and multilingual music contexts.

Figures

Figures reproduced from arXiv: 2502.10362 by the authors.

Figure 1
Figure 1. CLaMP 3 demonstrates robust cross-modal and cross-lingual generalization. Supervised alignment (solid arrows) links paired modalities, while emergent alignment (dashed arrows) bridges unaligned ones. A multilingual text encoder enables retrieval in languages unseen (grayed-out bubbles) during alignment. These capabilities position MIR as a critical bridge between music and language, supporting various applications b… view at source ↗
Figure 2
Figure 2. CLaMP 3 uses contrastive learning to align features across modalities. Sheet music and performance [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Language distribution of original and trans [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Country-wise distribution of music tracks in [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 6
Figure 6. Figure 6: The metadata generation prompt was used for constructing the M4-RAG dataset. This prompt outlines the [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]
Figure 7
Figure 7. Figure 7: Metadata examples from the M4-RAG and WikiMT-X datasets. The top section shows an entry for [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 8
Figure 8. Figure 8: a includes features from Text (back￾ground annotations), ABC notation, MIDI, and Audio. Each modality forms a distinct cluster, re￾flecting the inherent differences in how information is encoded. Notably, modalities closer to Text tend to perform better, aligning with …

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 8 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Text2Score: Generating Sheet Music From Textual Prompts

    cs.SD 2026-05 unverdicted novelty 7.0 of 10

    Text2Score turns text prompts into sheet music by having an LLM produce a bar-wise structural plan and a hierarchical decoder write ABC notation from that plan.

  2. DiffRhythm+: Controllable and Flexible Full-Length Song Generation with Preference Optimization

    eess.AS 2025-07 conditional novelty 6.0 of 10

    DiffRhythm+ improves full-length lyric-to-song generation via balanced data scaling, MuLan-based multimodal style control, and DPO fine-tuning guided by automated aesthetic scorers.

  3. Scaling Self-Supervised Representation Learning for Symbolic Piano Performance

    cs.SD 2025-06 conditional novelty 6.0 of 10

    Self-supervised pretraining on 60,000 hours of symbolic piano music produces a generative model and contrastive embeddings that beat leading baselines on continuation quality and several MIR classification benchmarks.

  4. Can Large Language Models Predict Audio Effects Parameters from Natural Language?

    cs.SD 2025-05 conditional novelty 6.0 of 10

    LLMs can predict equalizer and reverb parameters from natural language descriptions, and adding DSP features, DSP function code, and few-shot examples improves the predictions.

  5. SMART: Tuning a symbolic music generation system with an audio domain aesthetic reward

    cs.SD 2025-04 conditional novelty 6.0 of 10

    SMART uses an audio aesthetic reward to fine-tune a symbolic MIDI piano model, raising human enjoyment ratings by about 1.2 points in a small listening study while over-optimization collapses output diversity.

  6. Multimodal Video-to-Music Recommendation via Semantic Retrieval and Temporal Reranking

    cs.MM 2026-07 unverdicted novelty 5.0 of 10

    VTMR is a two-stage video-to-music recommender: joint audio-visual-text retrieval of candidates, then temporal-sequence reranking, lifting R@10 to 18.3 and matching commercial preference.

  7. TalkPlay-Tools: Conversational Music Recommendation with LLM Tool Calling

    cs.IR 2025-10 conditional novelty 5.0 of 10

    An LLM that plans tool calls — SQL, BM25, dense, and semantic-ID retrieval — yields small Hit@K gains over BM25-style baselines for conversational music recommendation on the synthetic TalkPlayData 2 benchmark.

  8. PianoBind: A Multimodal Joint Embedding Model for Pop-piano Music

    cs.SD 2025-09 conditional novelty 4.0 of 10

    PianoBind, a trimodal audio-MIDI-text embedding model trained on piano data, beats general-purpose music embedding models on pop-piano text-to-music retrieval benchmarks.

Reference graph

Works this paper leans on

36 extracted references · 13 canonical work pages · cited by 8 Pith papers

  1. [1]

    Andrea Agostinelli, Timo I Denk, Zal \'a n Borsos, Jesse Engel, Mauro Verzetti, Antoine Caillon, Qingqing Huang, Aren Jansen, Adam Roberts, Marco Tagliasacchi, et al. 2023. Musiclm: Generating music from text. arXiv preprint arXiv:2301.11325

    arXiv 2023

  2. [2]

    Jisheng Bai, Haohe Liu, Mou Wang, Dongyuan Shi, Wenwu Wang, Mark D Plumbley, Woon-Seng Gan, and Jianfeng Chen. 2024. Audiosetcaps: An enriched audio-caption dataset using automated generation pipeline with large audio and language models. arXiv preprint arXiv:2411.18953

    arXiv 2024

  3. [3]

    Lo \" c Barrault, Yu-An Chung, Mariano Cora Meglioli, David Dale, Ning Dong, Paul-Ambroise Duquenne, Hady Elsahar, Hongyu Gong, Kevin Heffernan, John Hoffman, et al. 2023. Seamlessm4t-massively multilingual & multimodal machine translation. arXiv preprint arXiv:2308.11596

    arXiv 2023

  4. [4]

    Dmitry Bogdanov, Minz Won, Philip Tovstogan, Alastair Porter, and Xavier Serra. 2019. The mtg-jamendo dataset for automatic music tagging. ICML

    work page 2019

  5. [6]

    Yi-Hui Chou, I Chen, Chin-Jui Chang, Joann Ching, Yi-Hsuan Yang, et al. 2021. Midibert-piano: large-scale pre-training for symbolic music understanding. arXiv preprint arXiv:2107.05223

    arXiv 2021

  6. [7]

    Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzm \' a n, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. https://doi.org/10.18653/V1/2020.ACL-MAIN.747 Unsupervised cross-lingual representation learning at scale . In Proceedings of the 58th Annual Meeting of the Association for Com...

    doi:10.18653/v1/2020.acl-main.747 2020

  7. [8]

    Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi, and Alexandre D \' e fossez. 2023. http://papers.nips.cc/paper\_files/paper/2023/hash/94b472a1842cd7c56dcb125fb2765fbd-Abstract-Conference.html Simple and controllable music generation . In Advances in Neural Information Processing Systems 36: Annual Conference on Neura...

    work page 2023

  8. [9]

    Seungheon Doh, Keunwoo Choi, Jongpil Lee, and Juhan Nam. 2023 a . https://doi.org/10.5281/ZENODO.10265311 Lp-musiccaps: Llm-based pseudo music captioning . In Proceedings of the 24th International Society for Music Information Retrieval Conference, ISMIR 2023, Milan, Italy, November 5-9, 2023 , pages 409--416

    doi:10.5281/zenodo.10265311 2023

Show all 36 references
  1. [10]

    Seungheon Doh, Minhee Lee, Dasaem Jeong, and Juhan Nam. 2024. https://doi.org/10.1109/ICASSP48485.2024.10446380 Enriching music descriptions with A finetuned-llm and metadata for text-to-music retrieval . In IEEE International Conference on Acoustics, Speech and Signal Process...

    2024

  2. [12]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    2024 arXiv

  3. [13]

    Lucas Ferreira and Jim Whitehead. 2019. http://archives.ismir.net/ismir2019/paper/000045.pdf Learning to generate music with sentiment . In Proceedings of the 20th International Society for Music Information Retrieval Conference, ISMIR 2019, Delft, The Netherlands, November 4-...

    2019

  4. [14]

    Gemmeke, Daniel P

    Jort F. Gemmeke, Daniel P. W. Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R. Channing Moore, Manoj Plakal, and Marvin Ritter. 2017. https://doi.org/10.1109/ICASSP.2017.7952261 Audio set: An ontology and human-labeled dataset for audio events . In 2017 IEEE International...

    2017

  5. [15]

    Rohit Girdhar, Alaaeldin El - Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. 2023. https://doi.org/10.1109/CVPR52729.2023.01457 Imagebind one embedding space to bind them all . In IEEE/CVF Conference on Computer Vision and Pattern Recog...

    2023

  6. [16]

    Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He

    Priya Goyal, Piotr Doll \' a r, Ross B. Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. 2017. https://arxiv.org/abs/1706.02677 Accurate, large minibatch SGD: training imagenet in 1 hour . CoRR, abs/1706.02677

    2017 arXiv

  7. [17]

    Qingqing Huang, Aren Jansen, Joonseok Lee, Ravi Ganti, Judith Yue Li, and Daniel P. W. Ellis. 2022. https://archives.ismir.net/ismir2022/paper/000067.pdf Mulan: A joint embedding of music audio and natural language . In Proceedings of the 23rd International Society for Music I...

    2022

  8. [18]

    Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tom \' a s Mikolov. 2017. https://doi.org/10.18653/V1/E17-2068 Bag of tricks for efficient text classification . In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics, ...

    2017 doi

  9. [19]

    u ttler, Mike Lewis, Wen - tau Yih, Tim Rockt \

    Patrick S. H. Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K \" u ttler, Mike Lewis, Wen - tau Yih, Tim Rockt \" a schel, Sebastian Riedel, and Douwe Kiela. 2020. https://proceedings.neurips.cc/paper/2020/hash/6b493230205f780e...

    2020

  10. [20]

    Dannenberg, Ruibo Liu, Wenhu Chen, Gus Xia, Yemin Shi, Wenhao Huang, Zili Wang, Yike Guo, and Jie Fu

    Yizhi Li, Ruibin Yuan, Ge Zhang, Yinghao Ma, Xingran Chen, Hanzhi Yin, Chenghao Xiao, Chenghua Lin, Anton Ragni, Emmanouil Benetos, Norbert Gyenge, Roger B. Dannenberg, Ruibo Liu, Wenhu Chen, Gus Xia, Yemin Shi, Wenhao Huang, Zili Wang, Yike Guo, and Jie Fu. 2024. https://open...

    2024

  11. [21]

    Ilya Loshchilov and Frank Hutter. 2019. https://openreview.net/forum?id=Bkg6RiCqY7 Decoupled weight decay regularization . In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 . OpenReview.net

    2019

  12. [22]

    Ilaria Manco, Benno Weck, Seungheon Doh, Minz Won, Yixiao Zhang, Dmitry Bogdanov, Yusong Wu, Ke Chen, Philip Tovstogan, Emmanouil Benetos, et al. 2023. The song describer dataset: a corpus of audio captions for music-and-language evaluation. arXiv preprint arXiv:2311.10057

    2023 arXiv

  13. [23]

    Jan Melechovsky, Abhinaba Roy, and Dorien Herremans. 2024. Midicaps--a large-scale midi dataset with text captions. arXiv preprint arXiv:2406.02255

    2024 arXiv

  14. [24]

    Diamos, Erich Elsen, David Garc \' a, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu

    Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory F. Diamos, Erich Elsen, David Garc \' a, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. 2018. https://openreview.net/forum?id=r1gs9JgRZ Mixed precision training . In 6th International C...

    2018

  15. [25]

    Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748

    2018 arXiv

  16. [26]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei - Jing Zhu. 2002. https://doi.org/10.3115/1073083.1073135 Bleu: a method for automatic evaluation of machine translation . In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, July 6-12, ...

    2002

  17. [27]

    Jan Retkowski, Jakub St e pniak, and Mateusz Modrzejewski. 2024. Frechet music distance: A metric for generative symbolic music evaluation. arXiv preprint arXiv:2412.07948

    2024 arXiv

  18. [28]

    Kihyuk Sohn. 2016. https://proceedings.neurips.cc/paper/2016/hash/6b180037abbebea991d8b1232f8a8ca9-Abstract.html Improved deep metric learning with multi-class n-pair loss objective . In Advances in Neural Information Processing Systems 29: Annual Conference on Neural Informat...

    2016

  19. [29]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. https://proceedings.neurips.cc/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html Attention is all you need . In Advances in Neural I...

    2017

  20. [30]

    Shangda Wu, Yashan Wang, Ruibin Yuan, Zhancheng Guo, Xu Tan, Ge Zhang, Monan Zhou, Jing Chen, Xuefeng Mu, Yuejie Gao, et al. 2024. Clamp 2: Multimodal music information retrieval across 101 languages using large language models. arXiv preprint arXiv:2410.13267

    2024 arXiv

  21. [31]

    Shangda Wu, Dingyao Yu, Xu Tan, and Maosong Sun. 2023 a . https://doi.org/10.5281/ZENODO.10265247 Clamp: Contrastive language-music pre-training for cross-modal symbolic music information retrieval . In Proceedings of the 24th International Society for Music Information Retrie...

    2023 doi

  22. [32]

    Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor Berg - Kirkpatrick, and Shlomo Dubnov. 2023 b . https://doi.org/10.1109/ICASSP49357.2023.10095969 Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation . In IEEE Internati...

    2023

  23. [33]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. 2024. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115

    2024 arXiv

  24. [34]

    Dannenberg, Wenhu Chen, Gus Xia, Wei Xue, Si Liu, Shi Wang, Ruibo Liu, Yike Guo, and Jie Fu

    Ruibin Yuan, Yinghao Ma, Yizhi Li, Ge Zhang, Xingran Chen, Hanzhi Yin, Le Zhuo, Yiqi Liu, Jiawen Huang, Zeyue Tian, Binyue Deng, Ningzhi Wang, Chenghua Lin, Emmanouil Benetos, Anton Ragni, Norbert Gyenge, Roger B. Dannenberg, Wenhu Chen, Gus Xia, Wei Xue, Si Liu, Shi Wang, Rui...

    2023

  25. [35]

    Mingliang Zeng, Xu Tan, Rui Wang, Zeqian Ju, Tao Qin, and Tie - Yan Liu. 2021. https://doi.org/10.18653/V1/2021.FINDINGS-ACL.70 Musicbert: Symbolic music understanding with large-scale pre-training . In Findings of the Association for Computational Linguistics: ACL/IJCNLP 2021...

    2021 doi

  26. [36]

    Haina Zhu, Yizhi Zhou, Hangting Chen, Jianwei Yu, Ziyang Ma, Rongzhi Gu, Wei Tan, and Xie Chen. 2025. Muq: Self-supervised music representation learning with mel residual vector quantization. arXiv preprint arXiv:2501.01108

    2025 arXiv

  27. [37]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  28. [38]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.