{"id":"3bf0f579-a6dd-4ed4-bc35-97655b243b7d","arxiv_id":"2502.07461","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A dataset of 362,000 Jamendo instrumental tracks pairs each song with a generated caption and imputed metadata fields, created with a retrieval-based local-LLM pipeline.","lead":"This paper releases JamendoMaxCaps, a music-caption dataset of over 362,000 instrumental tracks from Jamendo with generated captions and filled-in metadata. It also describes a retrieval-and-LLM pipeline that imputes missing genre, tempo, and tag fields, and reports small-scale evaluations of that imputation.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The objective imputation evaluation leaks the ground-truth target field into the retrieval query via Eq. (2), so the reported BERT-Score/BLEU gains may not transfer to genuinely missing metadata.","rationale":"The strongest_claim is that JamendoMaxCaps is a large-scale, freely licensed music-caption dataset with imputed metadata. The dataset artifact itself is genuine: 362k tracks, released code and data, and captions generated by Qwen2-Audio are valuable independent of imputation quality. The load-bearing condition for the imputation sub-claim is that the evaluation measures performance in the actual missing-metadata regime. It does not: Eq. (2) puts the ground-truth target field into the retrieval query, so the top-10 in-context examples are chosen with knowledge of the answer. This is a concrete, correctable flaw rather than an internal contradiction. The reader's weakest_assumption identifies exactly this leakage, and I agree with that reading. Because the existing verdict is CONDITIONAL and the requested fixes (field masking, variance reporting, larger subjective samples) are already the appropriate conditions, no verdict change is needed. The concern should not be escalated to rejection, since the dataset contribution stands and a masked re-evaluation could plausibly confirm the method; it should also not be downgraded, because the imputation claim is not yet well supported.","tokens_in":10377,"tokens_out":2721,"duration_ms":28062,"concrete_test":"Mask the target field in the metadata embedding for the query song in Eq. (2) and re-run the §IV-C protocol on the same 5,000 examples per field: set genres (or speed, vartags) to empty when computing F(S), while leaving other metadata intact; retrieve top 10, impute, and compare BERT-Score/BLEU with Table IV. If BERT-Score for genres falls from 0.58 toward the generic-example baseline 0.51, or the retrieval-vs-generic gap shrinks materially for speed or vartags, the reported imputation effectiveness is partly a leakage artifact.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central new method is metadata imputation, and its quantitative support is Table IV. In §IV-C, the authors select 5,000 examples per field (genres, speed, vartags) that have valid original values, then impute those fields and compare. The retrieval query in Eq. (2) is X(S) = λ1 P(M(S)) + λ2 F(S), where F(S) encodes the song's available metadata. For these evaluation songs, the field being imputed is present in the original metadata and therefore is included in F(S). The top-10 retrieved songs are thus likely to share that exact field, which makes the LLM's in-context inference substantially easier than in the real imputation setting, where the target field is absent. The generic-examples baseline does not control for this: it omits query-specific retrieval entirely, so the comparison measures leakage-augmented retrieval against no retrieval. The paper never states that the target field is masked before retrieval. Consequently, the BERT-Score and BLEU numbers in Table IV do not establish that the pipeline works when metadata is genuinely missing. The listening study is orthogonal, as it rates plausibility of imputed values rather than correctness against a withheld field.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces JamendoMaxCaps, a large-scale dataset of 362,238 instrumental tracks from the Jamendo platform, each paired with a caption generated by Qwen2-Audio and with metadata fields (genres, speed, vartags, etc.) imputed through a retrieval-based in-context learning pipeline. The retrieval system combines MERT audio features with metadata embeddings, retrieves top-k similar songs, and uses Llama-2 to infer missing metadata. The authors evaluate the imputation with BERT-Score/BLEU on 5,000 examples per field against a generic-examples baseline, and with a listening study rating plausibility of the imputed metadata. The dataset and code are publicly released. The central technical claims are the effectiveness of the retrieval-based imputation and the value of the resulting dataset as a public resource.","tokens_in":10559,"tokens_out":4848,"duration_ms":46136,"significance":"If the imputation results are valid, the dataset is a substantial contribution: it is an order of magnitude larger than MusicCaps, is freely licensed, and includes metadata that could benefit music-language tasks such as retrieval, representation learning, and generation. The paper is also praiseworthy for releasing code and data, and for describing the imputation pipeline in sufficient detail to be replicable. The main risk is that the quantitative validation of the imputation quality currently has a methodological flaw (evaluation leakage), and the retrieval weighting rests on a very small subjective study. These issues affect the strength of the core claim that the imputation works well, but they are addressable within the manuscript's scope.","major_comments":[{"comment":"The objective evaluation of metadata imputation leaks the target field into the retrieval query. For the 5,000 evaluation examples per field, the field being imputed is part of the metadata vector F(S) used in Eq. (2), so the top-10 retrieved songs are biased to share that very field with the query. The generic-examples baseline omits query-specific retrieval entirely, so the comparison in Table IV conflates genuine retrieval benefit with label leakage. The manuscript does not state that the target field is masked from F(S) before retrieval. As a result, the reported BERT-Score and BLEU gains do not establish that the pipeline transfers to the actual setting where the target field is missing. The authors should repeat the evaluation with the target field excluded from the retrieval features, or introduce a control that masks it, and report the corrected numbers.","section":"§IV-C, Eq. (2), Table IV"},{"comment":"The retrieval weighting λ1 = 0.6, λ2 = 0.4 is selected based on subjective ratings from only two participants on five songs. This is an ad-hoc basis for a hyperparameter that directly controls the quality of the in-context examples and, in turn, the imputation accuracy. The paper states that the difference was \"stark\" and therefore no further evaluation was conducted, but this does not justify the robustness of the chosen λ. The authors should either justify λ with a larger or more objective evaluation—for instance, by cross-validating imputation accuracy on the 5,000-example set—or explicitly discuss the sensitivity of the imputation results to λ.","section":"§IV-B, Table III"},{"comment":"The description of the MERT feature representation is internally inconsistent. Eq. (1) writes M(S) as an average over time frames with an unresolved index n, and the accompanying text says \"we take the average of all layers\" yet also states that M(S) ∈ R^{N×1024}. If M(S) is the result of averaging over layers, it should be 1024-dimensional, not N×1024; if it is a concatenation or stacking of per-layer averages, the projection P(.) and the subsequent concatenation with F(S) are not defined consistently. This ambiguity affects the reproducibility of the retrieval system, which is the core of the imputation method. Please clarify the exact form of M(S), the role of n in Eq. (1), and the dimensions involved in Eq. (2).","section":"§III-B, Eq. (1)–(2)"}],"minor_comments":[{"comment":"The acronym \"LLLM\" is used for \"local large language model\" in the abstract and elsewhere; the standard abbreviation is \"LLM\" or \"local LLM.\" Please fix the terminology for consistency.","section":"Abstract and throughout"},{"comment":"Eq. (2) has a misplaced parenthesis: X(S) = λ1 × (P(M(S)) + λ2 × F(S) is missing a closing parenthesis. The intended expression is presumably X(S) = λ1 P(M(S)) + λ2 F(S) with λ1 + λ2 = 1. Please correct the equation.","section":"§III-B, Eq. (2)"},{"comment":"There are a few reference-related inconsistencies: \"WaveCaps\" should be \"WavCaps\" (the cited dataset), and the sentence introducing \"MuLaMCap\" appears to attribute it to Noise2Music [10], which does not seem to be the correct source for that dataset. Please verify and correct the citations.","section":"§II, Related Work"},{"comment":"The paper does not report any evaluation of the quality of the generated captions themselves, despite captions being a primary component of the released dataset. Since Qwen2-Audio is an established model, the omission is understandable, but a brief sanity check (e.g., a small human rating or comparison with MusicCaps captions) would strengthen the resource claim.","section":"§III-A and §IV"},{"comment":"The listening study reports means and standard deviations, but the paper does not specify the number of songs rated per participant or how samples were selected. Adding these details would improve reproducibility.","section":"§IV-D, Table V"}],"recommendation":"major_revision","confidential_remarks":"The dataset is potentially a valuable community resource, and the manuscript is generally well written. The main concern is the objective evaluation leakage, which the authors can fix by masking the target field in the retrieval features. The λ selection issue is also fixable with a more thorough evaluation. I would be comfortable with acceptance after these revisions are made and reported. I also suggest that the authors explicitly acknowledge the limitations of the two-participant retrieval study in the main text."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe one thing to know: JamendoMaxCaps is a genuinely useful public resource—362k instrumental tracks with captions and partially imputed metadata, released on HuggingFace with code. If you work on music-language tasks, this is a meaningful expansion over MusicCaps. But the paper's central methodological claim—that retrieval-based in-context imputation works—is not well supported, because the objective evaluation leaks the target field into the retrieval query.\n\nWhat's new is the scale and the public release. The imputation pipeline is a fresh combination of known components (MERT audio features, flan-t5-encoded metadata, Llama-2 in-context learning), but each piece exists. The dataset itself is the contribution.\n\nThe soft spot is real. In §IV-C, for each of the 5,000 examples per field, the field being imputed is present in the original metadata, and that metadata is part of F(S) in Eq. (2). So the top-10 retrieved songs are likely to share the exact genre or speed value, making the LLM's inference easier than in the true missing-metadata setting. The generic-examples baseline uses random examples, not retrieval, so it doesn't control for this. The listening study rates plausibility, not correctness, so it doesn't rescue the claim. The λ tuning is based on two participants and five songs—thin, though they at least report it. No variance or significance is reported for Table IV.\n\nNone of this invalidates the dataset. The captions and imputed metadata are still useful training material, and the authors are candid about limitations like genre bias and local-LLM tradeoffs. But if you want to claim the imputation method works, you need to mask the target field during retrieval, report uncertainty, and enlarge the subjective samples.\n\nWho should read it: anyone in MIR or text-to-music looking for large-scale public data. It deserves a serious referee—the dataset should get into the ecosystem, but the evaluation needs fixing first.","headline":"A valuable public dataset, but the imputation evaluation leaks the target field into the retrieval query and does not support the central methodological claim.","tokens_in":11147,"tokens_out":2668,"would_cite":true,"duration_ms":23902,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"JamendoMaxCaps supplies 362,000 captioned instrumental tracks with imputed metadata.","keywords":["music-caption dataset","metadata imputation","music retrieval","in-context learning","large language model","music representation learning","text-to-music generation","multimodal music understanding"],"falsifier":"Run the imputation on 5,000 tracks with complete metadata while hiding their metadata from the retrieval query before comparing to the original values; if genre BERT-Score falls from around 0.58 toward the generic-example level (0.51), the claimed advantage for genuinely missing metadata is not supported.","tokens_in":10106,"feed_emoji":"🎵","tokens_out":6320,"duration_ms":55244,"temperature":0.7,"pith_summary":"JamendoMaxCaps is introduced as a large-scale music-caption resource: 362,238 freely licensed instrumental tracks from Jamendo, each paired with a natural-language caption generated by Qwen2-Audio and, where metadata was missing, imputed genre, speed, and variable tags. The paper argues this fills a real gap, since the most-used music-caption dataset, MusicCaps, has only about 5,000 examples and metadata in real collections is often incomplete. The imputation is done by retrieving similar songs with a hybrid audio-plus-metadata retriever and prompting a local 7-billion-parameter LLM with those songs as in-context examples. The authors validate the pipeline with retrieval-preference ratings, automatic comparison of imputed versus original metadata on 5,000 examples per field, and a 15-participant listening study, and they release the dataset publicly.","feed_headline":"New dataset adds captions and imputed metadata to 362,000 music tracks","feed_subtitle":"Freely licensed tracks gain captions, with retrieval-based LLM imputation filling missing genre, tempo, and mood tags.","key_machinery":"The load-bearing mechanism is the retrieval-imputation loop. A query vector is formed as $X(S) = \\lambda_1 P(M(S)) + \\lambda_2 F(S)$, where $M(S)$ is the averaged MERT audio feature, $P$ is a sparse random projection to 768 dimensions, $F(S)$ is the flan-t5 encoding of the available metadata, and $\\lambda_1 + \\lambda_2 = 1$. Cosine similarity on this vector retrieves the top-10 most similar songs; their metadata and captions are concatenated as in-context examples for Llama-2-7B, which infers the missing fields for the target caption. The weighting $\\lambda_1 = 0.6, \\lambda_2 = 0.4$, favoring musical features, was chosen after a small subjective evaluation.","core_discovery":"The central claim is that a publicly available corpus of over 362,000 instrumental audio tracks can be turned into a richer music-language dataset by combining a state-of-the-art audio captioner with retrieval-based metadata imputation. The paper reports that retrieval-based in-context examples consistently beat generic examples for imputing genre, speed, and variable tags, with BERT-Scores of 0.58, 0.73, and 0.38 respectively, and a listening study rating all imputed fields above the midpoint of a 7-point scale. The authors see JamendoMaxCaps as a resource for music retrieval, multimodal representation learning, and text-conditioned music generation.","pith_inferences":["A stricter test of the imputation claim would withhold metadata from the retrieval query for evaluation songs; the current setup includes original metadata in the query, so true missing-metadata performance may be lower.","The imputed labels are LLM-generated, so downstream models trained on this dataset may inherit the genre imbalance the paper notes, reinforcing Electronic and Rock at the expense of Jazz and Experimental.","The same retriever-plus-context-LLM recipe transfers naturally to other weakly annotated media, such as audio events or video, where captions are available but structured metadata is sparse.","One way to test caption quality directly would be to evaluate the Qwen2-Audio captions against human-written captions on a subset like SongDescriber, beyond measuring metadata agreement."],"forward_implications":["Music-language models can now train on roughly 70 times more audio-caption pairs than MusicCaps, all under free licenses.","Metadata coverage increases: after imputation, missing 'speed' entries disappear and underrepresented genres such as Techno and Jazz gain more examples.","The automatic comparison shows retrieval-based in-context examples outperform generic random examples on BERT-Score and BLEU for genres, speed, and vartags.","The dataset is positioned to support music retrieval, representation learning, and text-to-music generation without relying on remote, costly captioning APIs.","Imputed metadata lets downstream tasks use structured fields like genre, tempo, and mood that were previously absent for 261,935 partially annotated tracks."],"supporting_citations":[{"why":"Qwen2-Audio is the captioning model used to generate natural-language descriptions of each 30-second segment.","marker":"[4]"},{"why":"MERT provides the audio feature representation used in the retrieval system.","marker":"[15]"},{"why":"In-context learning survey supports the prompt design of feeding top-10 similar songs to the LLM.","marker":"[8]"},{"why":"MusicCaps defines the small-scale baseline the dataset aims to surpass.","marker":"[1]"},{"why":"WavCaps shows the scale of audio-caption data in general audio, contrasting with the music-specific gap.","marker":"[19]"},{"why":"LP-MusicCaps uses existing metadata to generate pseudo-captions; the paper builds on this while addressing incomplete metadata.","marker":"[7]"},{"why":"BERT-Score is the primary metric for semantic similarity between original and imputed metadata.","marker":"[27]"},{"why":"BLEU is used as the n-gram overlap metric for imputed metadata fields.","marker":"[22]"}],"fun_headline_variants":["362K music tracks gain captions and smart metadata imputation","JamendoMaxCaps: 362K tracks with captions and LLM-filled metadata","New dataset pairs 362K tracks with captions and imputed tags","Retrieval-based imputation boosts music-caption dataset quality","Large music-caption dataset uses retrieval to fill missing metadata"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation measures imputation quality on songs whose original metadata exists and is included in the retrieval query (Equation 2), so the reported accuracy may not reflect how the method performs when metadata is genuinely absent, which is the case the method is meant to handle.","fun_headline_variants_meta":{"raw":{"variants":["362K music tracks gain captions and smart metadata imputation","JamendoMaxCaps: 362K tracks with captions and LLM-filled metadata","New dataset pairs 362K tracks with captions and imputed tags","Retrieval-based imputation boosts music-caption dataset quality","Large music-caption dataset uses retrieval to fill missing metadata"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000154,"raw_usage":{"total_tokens":1149,"prompt_tokens":822,"completion_tokens":327,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":438,"completion_tokens_details":{"reasoning_tokens":234}},"tokens_in":438,"tokens_out":327,"duration_ms":3400,"temperature":1.0,"reasoning_tokens":234,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T12:40:09.604717+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the imputation on 5,000 tracks with complete metadata while hiding their metadata from the retrieval query before comparing to the original values; if genre BERT-Score falls from around 0.58 toward the generic-example level (0.51), the claimed advantage for genuinely missing metadata is not supported.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"MERT provides the audio feature representation used in the retrieval system."},{"cited_title":"Agostinelli, T","cited_arxiv_id":null,"evidence_quote":"MusicCaps defines the small-scale baseline the dataset aims to surpass."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"WavCaps shows the scale of audio-caption data in general audio, contrasting with the music-specific gap."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"LP-MusicCaps uses existing metadata to generate pseudo-captions; the paper builds on this while addressing incomplete metadata."},{"cited_title":"Papineni, S","cited_arxiv_id":null,"evidence_quote":"BLEU is used as the n-gram overlap metric for imputed metadata fields."}],"review_version":1}