Pith. sign in

REVIEW 3 major objections 5 minor 2 cited by

JamendoMaxCaps: A Large Scale Music-caption Dataset with Imputed Metadata

T0 review · 3 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read JamendoMaxCaps supplies 362,000 captioned instrumental tracks with imputed metadata.

desk verdict A valuable public dataset, but the imputation evaluation leaks the target field into the retrieval query and does not support the central methodological claim. read the letter →

arxiv 2502.07461 v2 pith:DB45ZUNP submitted 2025-02-11 cs.SD cs.AI

Abhinaba Roy , Renhang Liu , Tongyu Lu , Dorien Herremans This is my paper · ORCID
classification cs.SDcs.AI
keywords music-captiondatasetmetadataimputationmusicretrievalin-contextlearninglargelanguagemodelrepresentationtext-to-musicgenerationmultimodalunderstanding
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

JamendoMaxCaps is introduced as a large-scale music-caption resource: 362,238 freely licensed instrumental tracks from Jamendo, each paired with a natural-language caption generated by Qwen2-Audio and, where metadata was missing, imputed genre, speed, and variable tags. The paper argues this fills a real gap, since the most-used music-caption dataset, MusicCaps, has only about 5,000 examples and metadata in real collections is often incomplete. The imputation is done by retrieving similar songs with a hybrid audio-plus-metadata retriever and prompting a local 7-billion-parameter LLM with those songs as in-context examples. The authors validate the pipeline with retrieval-preference ratings, automatic comparison of imputed versus original metadata on 5,000 examples per field, and a 15-participant listening study, and they release the dataset publicly.

What carries the argument

The load-bearing mechanism is the retrieval-imputation loop. A query vector is formed as $X(S) = \lambda_1 P(M(S)) + \lambda_2 F(S)$, where $M(S)$ is the averaged MERT audio feature, $P$ is a sparse random projection to 768 dimensions, $F(S)$ is the flan-t5 encoding of the available metadata, and $\lambda_1 + \lambda_2 = 1$. Cosine similarity on this vector retrieves the top-10 most similar songs; their metadata and captions are concatenated as in-context examples for Llama-2-7B, which infers the missing fields for the target caption. The weighting $\lambda_1 = 0.6, \lambda_2 = 0.4$, favoring musical features, was chosen after a small subjective evaluation.

What would settle it

Run the imputation on 5,000 tracks with complete metadata while hiding their metadata from the retrieval query before comparing to the original values; if genre BERT-Score falls from around 0.58 toward the generic-example level (0.51), the claimed advantage for genuinely missing metadata is not supported.

Watch

Extended reading notes

Core claim

The central claim is that a publicly available corpus of over 362,000 instrumental audio tracks can be turned into a richer music-language dataset by combining a state-of-the-art audio captioner with retrieval-based metadata imputation. The paper reports that retrieval-based in-context examples consistently beat generic examples for imputing genre, speed, and variable tags, with BERT-Scores of 0.58, 0.73, and 0.38 respectively, and a listening study rating all imputed fields above the midpoint of a 7-point scale. The authors see JamendoMaxCaps as a resource for music retrieval, multimodal representation learning, and text-conditioned music generation.

Load-bearing premise

The evaluation measures imputation quality on songs whose original metadata exists and is included in the retrieval query (Equation 2), so the reported accuracy may not reflect how the method performs when metadata is genuinely absent, which is the case the method is meant to handle.

Editorial extensions

If this is right

  • Music-language models can now train on roughly 70 times more audio-caption pairs than MusicCaps, all under free licenses.
  • Metadata coverage increases: after imputation, missing 'speed' entries disappear and underrepresented genres such as Techno and Jazz gain more examples.
  • The automatic comparison shows retrieval-based in-context examples outperform generic random examples on BERT-Score and BLEU for genres, speed, and vartags.
  • The dataset is positioned to support music retrieval, representation learning, and text-to-music generation without relying on remote, costly captioning APIs.
  • Imputed metadata lets downstream tasks use structured fields like genre, tempo, and mood that were previously absent for 261,935 partially annotated tracks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A stricter test of the imputation claim would withhold metadata from the retrieval query for evaluation songs; the current setup includes original metadata in the query, so true missing-metadata performance may be lower.
  • The imputed labels are LLM-generated, so downstream models trained on this dataset may inherit the genre imbalance the paper notes, reinforcing Electronic and Rock at the expense of Jazz and Experimental.
  • The same retriever-plus-context-LLM recipe transfers naturally to other weakly annotated media, such as audio events or video, where captions are available but structured metadata is sparse.
  • One way to test caption quality directly would be to evaluate the Qwen2-Audio captions against human-written captions on a subset like SongDescriber, beyond measuring metadata agreement.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces JamendoMaxCaps, a large-scale dataset of 362,238 instrumental tracks from the Jamendo platform, each paired with a caption generated by Qwen2-Audio and with metadata fields (genres, speed, vartags, etc.) imputed through a retrieval-based in-context learning pipeline. The retrieval system combines MERT audio features with metadata embeddings, retrieves top-k similar songs, and uses Llama-2 to infer missing metadata. The authors evaluate the imputation with BERT-Score/BLEU on 5,000 examples per field against a generic-examples baseline, and with a listening study rating plausibility of the imputed metadata. The dataset and code are publicly released. The central technical claims are the effectiveness of the retrieval-based imputation and the value of the resulting dataset as a public resource.

Significance. If the imputation results are valid, the dataset is a substantial contribution: it is an order of magnitude larger than MusicCaps, is freely licensed, and includes metadata that could benefit music-language tasks such as retrieval, representation learning, and generation. The paper is also praiseworthy for releasing code and data, and for describing the imputation pipeline in sufficient detail to be replicable. The main risk is that the quantitative validation of the imputation quality currently has a methodological flaw (evaluation leakage), and the retrieval weighting rests on a very small subjective study. These issues affect the strength of the core claim that the imputation works well, but they are addressable within the manuscript's scope.

major comments (3)
  1. [§IV-C, Eq. (2), Table IV] The objective evaluation of metadata imputation leaks the target field into the retrieval query. For the 5,000 evaluation examples per field, the field being imputed is part of the metadata vector F(S) used in Eq. (2), so the top-10 retrieved songs are biased to share that very field with the query. The generic-examples baseline omits query-specific retrieval entirely, so the comparison in Table IV conflates genuine retrieval benefit with label leakage. The manuscript does not state that the target field is masked from F(S) before retrieval. As a result, the reported BERT-Score and BLEU gains do not establish that the pipeline transfers to the actual setting where the target field is missing. The authors should repeat the evaluation with the target field excluded from the retrieval features, or introduce a control that masks it, and report the corrected numbers.
  2. [§IV-B, Table III] The retrieval weighting λ1 = 0.6, λ2 = 0.4 is selected based on subjective ratings from only two participants on five songs. This is an ad-hoc basis for a hyperparameter that directly controls the quality of the in-context examples and, in turn, the imputation accuracy. The paper states that the difference was "stark" and therefore no further evaluation was conducted, but this does not justify the robustness of the chosen λ. The authors should either justify λ with a larger or more objective evaluation—for instance, by cross-validating imputation accuracy on the 5,000-example set—or explicitly discuss the sensitivity of the imputation results to λ.
  3. [§III-B, Eq. (1)–(2)] The description of the MERT feature representation is internally inconsistent. Eq. (1) writes M(S) as an average over time frames with an unresolved index n, and the accompanying text says "we take the average of all layers" yet also states that M(S) ∈ R^{N×1024}. If M(S) is the result of averaging over layers, it should be 1024-dimensional, not N×1024; if it is a concatenation or stacking of per-layer averages, the projection P(.) and the subsequent concatenation with F(S) are not defined consistently. This ambiguity affects the reproducibility of the retrieval system, which is the core of the imputation method. Please clarify the exact form of M(S), the role of n in Eq. (1), and the dimensions involved in Eq. (2).
minor comments (5)
  1. [Abstract and throughout] The acronym "LLLM" is used for "local large language model" in the abstract and elsewhere; the standard abbreviation is "LLM" or "local LLM." Please fix the terminology for consistency.
  2. [§III-B, Eq. (2)] Eq. (2) has a misplaced parenthesis: X(S) = λ1 × (P(M(S)) + λ2 × F(S) is missing a closing parenthesis. The intended expression is presumably X(S) = λ1 P(M(S)) + λ2 F(S) with λ1 + λ2 = 1. Please correct the equation.
  3. [§II, Related Work] There are a few reference-related inconsistencies: "WaveCaps" should be "WavCaps" (the cited dataset), and the sentence introducing "MuLaMCap" appears to attribute it to Noise2Music [10], which does not seem to be the correct source for that dataset. Please verify and correct the citations.
  4. [§III-A and §IV] The paper does not report any evaluation of the quality of the generated captions themselves, despite captions being a primary component of the released dataset. Since Qwen2-Audio is an established model, the omission is understandable, but a brief sanity check (e.g., a small human rating or comparison with MusicCaps captions) would strengthen the resource claim.
  5. [§IV-D, Table V] The listening study reports means and standard deviations, but the paper does not specify the number of songs rated per participant or how samples were selected. Adding these details would improve reproducibility.

Circularity Check

1 steps flagged · score 5.0 of 10

The objective imputation evaluation leaks the ground-truth target field into the retrieval query via Eq. (2), so the Table IV gains may not transfer to genuinely missing metadata; the pipeline itself is not circular by construction.

  1. other [Section IV-C (Objective Evaluation), building on Section III-B Eq. (2) and the definition of F(S).]
    "To evaluate the accuracy of the metadata imputation, we first identify the examples that have valid entries in the original metadata for the ‘genres’, ‘speed’, and ‘vartags’ fields. […] For the metadata features, as the dataset contains partially available metadata, we first encode the existing metadata fields using the Google flan-t5 model. […] X (S) = λ1 × (P(M(S)) +λ2 × F(S)"

    On the 5,000 evaluation examples per field, the field being imputed has a valid original entry, so it is among the ‘existing metadata fields’ encoded into F(S) in Eq. (2). The query vector X(S) therefore contains the ground-truth value against which the imputation is scored, and cosine-similarity retrieval (Eq. 3) selects the top-10 in-context examples using that same field. The LLM then conditions on metadata/captions of songs already biased to share the target field, making the retrieved-example imputation easier than the real missing-metadata setting. The generic-examples baseline uses no retrieval at all, so the comparison in Table IV measures leakage-augmented retrieval versus no retrieval rather than imputation of genuinely absent fields.

full rationale

The core dataset construction — 362k Jamendo tracks, Qwen2-Audio captions, and an LLM imputation pipeline that does not read the missing field at inference time — is not circular. The central quantitative support for the imputation contribution, however, is partially circular: in Section IV-C the evaluation selects tracks whose original target field is present, and Section III-B defines F(S) as encoding all existing metadata, so that field enters the Eq. (2) query used to retrieve the in-context examples the model is scored on. No load-bearing self-citation chain was found; citations to the authors' own prior work are contextual. The result is not equivalent to its inputs by definition, but the headline imputation metrics are inflated by construction for the evaluated subset, giving a partial-circularity score of 5.

Assumptions & free parameters 4 free parameters · 6 assumptions · 0 invented entities

The central claims rest on several unverified modeling assumptions and a small set of tuned hyperparameters. The most consequential are the use of MERT features for musical similarity, trust in Qwen2-Audio captions, and the use of original Jamendo metadata as ground truth. The retrieval weighting and context size are hand-chosen with limited validation. No new physical or conceptual entities are introduced.

free parameters (4)
  • lambda_1 / lambda_2 = 0.6 / 0.4
    Weights for combining audio and metadata features in Equation 2, chosen from a subjective evaluation with 5 songs and 2 raters (Table III).
  • top_k similar songs = 10
    Number of in-context examples used for imputation; no ablation is provided to justify this value.
  • maximum caption tokens = 128
    Set to keep roughly 90% of generated captions within length; affects caption detail but is not central.
  • minimum retained segment length = 15 seconds
    Final segments shorter than 15 seconds are discarded based on a quality judgment; affects coverage of longer tracks.
assumptions (6)
  • domain assumption MERT layer-averaged features capture sufficient musical similarity for metadata imputation
    Used in Equations 1-3; the only support is the two-participant retrieval evaluation.
  • domain assumption Qwen2-Audio captions are accurate enough to serve as a basis for metadata imputation
    Caption quality is not evaluated in the paper; the model is treated as established.
  • domain assumption Original Jamendo metadata fields are valid ground truth for evaluation
    The objective evaluation in Section IV-C compares imputed values to original metadata without auditing those originals.
  • domain assumption In-context learning from ten similar songs transfers metadata patterns to the target song
    Core mechanism of Algorithm 1; evidence is indirect via BERT-Score and BLEU.
  • ad hoc to paper Two-participant Likert ratings on five songs are sufficient to select the retrieval weighting lambda
    Section IV-B and Table III; no statistical testing or additional participants were used.
  • domain assumption Cosine similarity on concatenated projected features is an appropriate retrieval metric
    Equation 3; no comparison to other retrieval metrics is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of JamendoMaxCaps: A Large Scale Music-caption Dataset with Imputed Metadata." pith.science (2026). https://pith.science/paper/DB45ZUNP

@misc{pith2026250207461,
  author       = {Pith},
  title        = {Pith review of: JamendoMaxCaps: A Large Scale Music-caption Dataset with Imputed Metadata},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DB45ZUNP}},
  note         = {Machine review of arXiv:2502.07461}
}
read the original abstract

We introduce JamendoMaxCaps, a large-scale music-caption dataset featuring over 362,000 freely licensed instrumental tracks from the renowned Jamendo platform. The dataset includes captions generated by a state-of-the-art captioning model, enhanced with imputed metadata. We also introduce a retrieval system that leverages both musical features and metadata to identify similar songs, which are then used to fill in missing metadata using a local large language model (LLLM). This approach allows us to provide a more comprehensive and informative dataset for researchers working on music-language understanding tasks. We validate this approach quantitatively with five different measurements. By making the JamendoMaxCaps dataset publicly available, we provide a high-quality resource to advance research in music-language understanding tasks such as music retrieval, multimodal representation learning, and generative music models.

Figures

Figures reproduced from arXiv: 2502.07461 by the authors.

Figure 1
Figure 1. Overview of our proposed pipeline for the creation of the dataset. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The metadata imputation process. described earlier. We then construct an in-context examples by concatenating the metadata and captions of these top 10 similar songs. In-context learning has proven to be highly effective in empowering large language models to generate meaningful and relevant outputs [8]. We leverage this approach by prompting the language model with these in-context exam￾ples and provide the caption… view at source ↗
Figure 3
Figure 3. Distribution of genres in original and imputed metadata. imputation process enhances the granularity of the metadata by introducing additional categories, including “Fast”, “Slow”, “Very Fast”, and “Moderate”, which were not initially present significantly in the original data. While “Medium” remains the predominant category, the imputed metadata offers a more nuanced distribution, capturing a broader range of tempo… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Distribution of speed in original and imputed metadata [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Distribution of variable tags in original and imputed metadata. features and metadata features as outlined in Equation 2. We vary the values of λ1 and λ2 in four distinct ways: • λ1 = 0; λ2 = 1 – Only rely on metadata information • λ1 = 1; λ2 = 0 – Only rely on musical…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Revisiting Audio-language Pretraining for Learning General-purpose Audio Representation

    eess.AS 2025-11 conditional novelty 6.0 of 10

    A 10.7M-pair audio-caption corpus and systematic comparison show contrastive pretraining is more data-efficient while captioning scales better, and supervised initialization yields diminishing returns.

  2. Auto-Regressive vs Flow-Matching: a Comparative Study of Modeling Paradigms for Text-to-Music Generation

    cs.SD 2025-06 conditional novelty 6.0 of 10

    Under matched training conditions, auto-regressive models slightly outperform flow-matching on music quality and temporal control, while flow-matching offers faster inference and better inpainting flexibility.

Reference graph

Works this paper leans on

27 extracted references · 21 canonical work pages · cited by 2 Pith papers

  1. [1]

    Agostinelli, T

    A. Agostinelli, T. I. Denk, Z. Borsos, J. Engel, M. Verzetti, A. Caillon, Q. Huang, A. Jansen, A. Roberts, M. Tagliasacchi, M. Sharifi, N. Zeghi- dour, and C. Frank. Musiclm: Generating music from text, 2023

    work page 2023

  2. [2]

    Bhandari, A

    K. Bhandari, A. Roy, K. Wang, G. Puri, S. Colton, and D. Herremans. Text2midi: Generating symbolic music from captions. In Proc. of AAAI, Philadelphia, US, 2025

    work page 2025

  3. [3]

    S. Chen, Y . Wu, C. Wang, S. Liu, D. Tompkins, Z. Chen, W. Che, X. Yu, and F. Wei. Beats: Audio pre-training with acoustic tokenizers. In Proc. of the 40th Int. Conf. on Machine Learning (ICML), Honolulu, USA , volume 202 of Proc. of Machine Learning Research (PMLR) , 2023

    work page 2023

  4. [4]

    Y . Chu, J. Xu, Q. Yang, H. Wei, X. Wei, Z. Guo, Y . Leng, Y . Lv, J. He, J. Lin, et al. Qwen2-audio technical report. arXiv:2407.10759, 2024

    arXiv 2024

  5. [5]

    Civit, V

    M. Civit, V . Drai-Zerbib, D. Lizcano, and M. Escalona. Sunocaps: A novel dataset of text-prompt based ai-generated music with emotion annotations. Data in Brief , 55:110743, 2024

    work page 2024

  6. [6]

    Copet, F

    J. Copet, F. Kreuk, I. Gat, T. Remez, D. Kant, G. Synnaeve, Y . Adi, and A. D ´efossez. Simple and controllable music generation. In Thirty- seventh Conf. on Neural Information Processing Systems , 2023

    work page 2023

  7. [7]

    S. Doh, K. Choi, J. Lee, and J. Nam. Lp-musiccaps: Llm-based pseudo music captioning, 2023

    work page 2023

  8. [8]

    Q. Dong, L. Li, D. Dai, C. Zheng, J. Ma, R. Li, H. Xia, J. Xu, Z. Wu, T. Liu, et al. A survey on in-context learning. arXiv:2301.00234, 2022

    arXiv 2022

Show all 27 references
  1. [9]

    Elizalde, S

    B. Elizalde, S. Deshmukh, M. A. Ismail, and H. Wang. Clap: Learning audio concepts from natural language supervision, 2023

    2023

  2. [10]

    Huang, D

    Q. Huang, D. S. Park, T. Wang, T. I. Denk, A. Ly, N. Chen, Z. Zhang, Z. Zhang, J. Yu, C. Frank, et al. Noise2music: Text-conditioned music generation with diffusion models. arXiv:2302.03917, 2023

    2023 arXiv

  3. [11]

    J. Kang, S. Poria, and D. Herremans. Video2music: Suitable music generation from videos using an affective multimodal transformer model. Expert Systems with Applications , 249:123640, 2024

    2024

  4. [12]

    Kim and T

    D. Kim and T. Kim. Missing modality prediction for unpaired multi- modal learning via joint embedding of unimodal models. In European Conf. on Computer Vision , pages 171–187. Springer, 2024

    2024

  5. [13]

    J. Lee, K. Lee, J. Park, J. Park, and J. Nam. Deep content-user embedding model for music recommendation. arXiv:1807.06786, 2018

    2018 arXiv

  6. [14]

    J. Li, D. Li, S. Savarese, and S. Hoi. BLIP-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. In Proc. of the 40 th Int. Conf. on Machine Learning (ICML), Honolulu, Hawaii, USA , volume 202 of Proc. of Machine Learning Research...

    2023

  7. [15]

    Y . Li, R. Yuan, G. Zhang, Y . Ma, X. Chen, H. Yin, C. Xiao, C. Lin, A. Ragni, E. Benetos, N. Gyenge, R. B. Dannenberg, R. Liu, W. Chen, G. Xia, Y . Shi, W. Huang, Z. Wang, Y . Guo, and J. Fu. MERT: acoustic music understanding model with large-scale self-supervised training. ...

    2024

  8. [16]

    R. Likert. A technique for the measurement of attitudes. Archives of Psychology, 1932

    1932

  9. [17]

    Manco, B

    I. Manco, B. Weck, S. Doh, M. Won, Y . Zhang, D. Bogdanov, Y . Wu, K. Chen, et al. The song describer dataset: a corpus of audio captions for music-and-language evaluation. arXiv:2311.10057, 2023

    2023 arXiv

  10. [18]

    X. Mei, C. Meng, H. Liu, Q. Kong, T. Ko, C. Zhao, M. D. Plumb- ley, Y . Zou, and W. Wang. Wavcaps: A chatgpt-assisted weakly- labelled audio captioning dataset for audio-language multimodal re- search. IEEE/ACM Trans. Audio Speech Lang. Process. , 2024

    2024

  11. [19]

    X. Mei, C. Meng, H. Liu, Q. Kong, T. Ko, C. Zhao, M. D. Plumb- ley, Y . Zou, and W. Wang. WavCaps: A ChatGPT-assisted weakly- labelled audio captioning dataset for audio-language multimodal re- search. IEEE/ACM Trans. Audio Speech Lang. Process. , pages 1–15, 2024

    2024

  12. [20]

    Melechovsky, Z

    J. Melechovsky, Z. Guo, D. Ghosal, N. Majumder, D. Herremans, and S. Poria. Mustango: Toward controllable text-to-music generation. In Proc. of the 2024 Conf. of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Language Technologies , pages ...

    2024

  13. [21]

    Melechovsky, A

    J. Melechovsky, A. Roy, and D. Herremans. Midicaps–a large-scale midi dataset with text captions. In Proc. of ISMIR , 2024

    2024

  14. [22]

    Papineni, S

    K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu. Bleu: a method for automatic evaluation of machine translation. In Proc. of the 40th annual meeting of the Association for Computational Linguistics , pages 311– 318, 2002

    2002

  15. [23]

    Radford, J

    A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever. Robust speech recognition via large-scale weak supervision. In Proc. of the 40 th Int. Conf. on Machine Learning (ICML), Honolulu, USA, volume 202 of Proc. of Machine Learning Research (PMLR) , 2023

    2023

  16. [24]

    G. Stoet. Psytoolkit: A software package for programming psychological experiments using linux. Behav. Res. Methods , 42:1096–1104, 2010

    2010

  17. [25]

    C. Tang, W. Yu, G. Sun, X. Chen, T. Tan, W. Li, L. Lu, Z. MA, and C. Zhang. SALMONN: Towards generic hearing abilities for large language models. In The 12th Int. Conf. on Learning Representations , 2024

    2024

  18. [26]

    Y . Wu, K. Chen, T. Zhang, Y . Hui, T. Berg-Kirkpatrick, and S. Dubnov. Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation. In IEEE Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP) , pages 1–5. IEEE, 2023

    2023

  19. [27]

    Zhang, V

    T. Zhang, V . Kishore, F. Wu, K. Q. Weinberger, and Y . Artzi. Bertscore: Evaluating text generation with bert. arXiv:1904.09675, 2019

    1904 arXiv

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.