{"id":"fd5f9b16-8740-4ee5-bad5-f1ad24091a50","arxiv_id":"2509.06936","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A new benchmark using expert musicological annotations shows that no single music AI encoder wins across all tagging tasks.","lead":"This paper creates a music autotagging benchmark from the expert-annotated MGPHot dataset, adding YouTube audio links, a standard train/test split, and precomputed audio embeddings from seven AI models. It compares these models on expert versus generic tags and finds that the best model depends heavily on the dataset and tag category.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Audio identity in MGPHot is unverified: 43.57% of YouTube links are unofficial, and no fingerprint check supports the central MGPHot ranking comparisons.","rationale":"The reader's weakest_assumption identifies the same load-bearing concern, and I agree. The reason this is load-bearing rather than a minor data-quality issue is that all MGPHot numbers in Table 3/Figure 3 are produced by feeding these unverified YouTube downloads into encoders. If mismatches are non-uniform across decades or upload types, the dataset-level ranking shifts (MAEST best on generic, CLAP/MERT best on MGPHot) could be an artifact of audio version or quality rather than annotation semantics. The paper does have real strengths: it releases URLs, download/rebuild scripts, MD5 checksums, and precomputed embeddings; the frozen-encoder protocol is standard; the split constraints (stratified by descriptors/year, disjoint main artists) are sensible. The concern is therefore a condition on the benchmark's validity, not a refutation. I would keep the verdict CONDITIONAL: the authors should add an audio fingerprint audit or a robustness analysis on official-only links before the benchmark is treated as canonical. I also noticed a smaller surface inconsistency: the text says seven encoders throughout, but Table 2/3 and the reference list enumerate six; this should be corrected but does not change the central conclusion.","tokens_in":7856,"tokens_out":5570,"duration_ms":67740,"concrete_test":"Download a stratified random sample of 500 linked videos (oversample unofficial and pre-1980 tracks to capture likely mismatch cases), fingerprint the audio with an external service such as ACRCloud or compare Chromaprint fingerprints against reference masters from a catalog API, and compute exact-match rate overall and by official/unofficial and decade. Then re-run the MGPHot-tag and MGPHot-reg probes on the confirmed-matching subset and compare the ranking/winner to Table 3. If the match rate is below ~95% or the winner/margins change on the confirmed subset, the reported MGPHot rankings are not reliable.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 2 links every MGPHot track to a YouTube video using title/artist regex (72.91%), then two Qwen2.5 LLM passes over titles/descriptions (22.86% + 3.47%), with manual review only for the residual 163 tracks. No step verifies that the audio content corresponds to the exact recording annotated in MGPHot. Since 56.43% of accepted links are official, 43.57% are unofficial; these may be covers, live takes, re-recordings, remasters, or unrelated videos. The MGPHot labels are fine-grained musicological attributes ('Vocal Grittiness', 'Harmonic sophistication', 'Aural Intensity') that are properties of a particular performance/recording, not just of a song title. If a non-trivial fraction of links point to wrong versions, the MGPHot columns in Table 3 and Figure 3 are noisy or biased. Bias is plausible because match quality likely correlates with track age/popularity and upload type; both correlate with the acoustic content being evaluated. The paper reports no audio fingerprint check, no manual audit of automated matches, and no sensitivity analysis excluding or down-weighting unofficial links. Without this, the central claim that MGPHot reveals different model rankings than generic tag datasets is not yet established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a new music autotagging benchmark built on the MGPHot dataset of expert musicological annotations. It links all 21,320 MGPHot tracks to YouTube audio using a regex/LLM matching pipeline, defines a canonical train/val/test split with artist-disjoint stratification, and releases precomputed embeddings for seven audio encoders. The authors evaluate these encoders on MGPHot (regression and binarized classification) and on two generic tag datasets (MTG-Jamendo, MagnaTagATune), reporting per-category and per-tag performance. The central finding is that no single encoder leads across all datasets and categories, and that rankings shift between expert-annotated and generic tag datasets.","tokens_in":8177,"tokens_out":3416,"duration_ms":39878,"significance":"If the audio-identity concern is resolved, this is a valuable contribution: it extends MGPHot into a reusable audio benchmark, provides a carefully constructed split, and releases code, metadata, and embeddings that support reproducible evaluation. The probe protocol (frozen encoder + MLP, five seeds, significance tests) is rigorous, and the per-category analysis offers a more granular view than previous autotagging benchmarks. The finding that model rankings differ across label types would be an important caution for the MIR community. The release of precomputed features and reconstruction scripts is a concrete reproducibility asset.","major_comments":[{"comment":"The audio identity of the YouTube links is not verified. The pipeline matches tracks by title/artist regex (72.91%) and LLM checks over titles/descriptions (22.86% + 3.47%), but never checks whether the audio content corresponds to the exact MGPHot recording. Since 43.57% of accepted links are unofficial uploads, they may contain covers, live performances, remasters, or unrelated audio. MGPHot annotations are recording-specific (e.g., 'Vocal Grittiness', 'Harmonic sophistication'), so wrong audio would directly invalidate the MGPHot results in Table 3 and Figure 3. The paper should provide an audio fingerprinting check or, at minimum, a manual audit of a sample and a sensitivity analysis restricted to official uploads. Without this, the central claim of ranking shifts between MGPHot and generic datasets is not established.","section":"Section 2"},{"comment":"The mapping of MTG-Jamendo and MagnaTagATune tags into categories (Genre, Instrument, Mood/Theme, etc.) is not described. The text states that the vocabularies are 'mapped into higher-level musical categories,' but the mapping itself is not given, nor is it mentioned as released. This makes the per-category heatmaps in Figure 3 non-reproducible and potentially arbitrary if categories were chosen after seeing results. The authors should release the tag-to-category mapping and justify the assignment (e.g., by using existing taxonomic groupings). This is load-bearing for the per-category ranking shifts that support the paper's conclusions.","section":"Section 5 / Figure 3"}],"minor_comments":[{"comment":"The criterion for 'official' uploads is not defined. It would be helpful to state whether 'official' is determined by channel type (e.g., artist topic, verified label) and how this was verified.","section":"Section 2"},{"comment":"The split stratification description is ambiguous: 'computed over all label bins' could refer to the 174 binarized tags or to continuous value bins. Please clarify whether stratification is performed on the continuous descriptors or on the binarized tags, and define the binning used.","section":"Section 3"},{"comment":"The color scale for MGPHot-reg uses RMSE (lower is better) while the left panels use MAP (higher is better). This is stated in the caption, but the distinct scales and directions should be made more visually explicit in the figure itself.","section":"Figure 3"},{"comment":"The notation 'bold and underlined' is explained, but the table also uses gray background for top-3. Please ensure the caption fully disambiguates these markings, especially when a cell is bold but not underlined.","section":"Table 3"},{"comment":"Early stopping with patience 50 is reported, but the number of training epochs or the validation metric used for early stopping is not given. This is a minor reproducibility detail.","section":"Section 4"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know about this paper. First, the resource is real: it provides the first canonical train/val/test split for MGPHot, a curated set of YouTube links with retrievable audio, and precomputed embeddings for seven encoders. That is useful work, and the community will likely adopt it. Second, the entire benchmark rests on the assumption that the YouTube audio actually is the recording described in the MGPHot annotations. The authors matched tracks to videos using textual metadata only, and 43.57% of the accepted links are unofficial uploads. No audio fingerprinting or manual audit was done. If a different version or a cover got linked, the expert annotations describe a different performance than the audio being evaluated, which could bias the reported rankings, especially since match quality likely correlates with song age and popularity.\n\nWhat the paper does well: the split-generation protocol is careful — disjoint main artists, stratification by year and official-uploader balance, with a target of <2% difference in label distributions. The evaluation itself follows best practice: frozen encoders, a simple probe, five seeds, and significance tests. The finding that no single encoder wins across all settings is plausible and worth stating. The self-citation overlap (MGPHot, OMAR-RQ from the same lab) is not circular because the annotations are externally sourced and the comparison is grounded on two independent generic tag datasets.\n\nThe soft spots: the audio-identity issue is the load-bearing one, and the paper does not address it in the limitations. Also, no diagnostics of the final split are reported — we do not see how close the stratified descriptor distributions actually got to the 2% target. That is a minor omission. The good news is that the release includes code and embeddings, so a sensitivity analysis on official vs. unofficial uploads is feasible in a revision.\n\nThe bottom line: this paper deserves a serious referee. It is a solid, useful benchmark paper with one clear, fixable weakness. I would send it to review, and I would ask for either an audio-identity check or a quantitative estimate of the mismatch risk before accepting.","headline":"A genuinely useful benchmark resource for MIR that is currently held back by an unverified audio-identity assumption the authors need to address.","tokens_in":8571,"tokens_out":2450,"would_cite":true,"duration_ms":27974,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Turning expert-annotated MGPHot (21,320 Billboard tracks) into an audio benchmark, the paper shows no single encoder leads everywhere: rankings shift between generic tags and expert musical attributes.","keywords":["music autotagging","MGPHot","expert musicological annotations","audio representation learning","benchmark evaluation","music information retrieval","Billboard Hot 100"],"falsifier":"Re-run the paper's probe protocol on only the 56.43% of MGPHot tracks that come from official uploads and re-rank the seven encoders. The central claim — that rankings shift between generic and expert-annotated benchmarks — should survive on official-only audio; if CLAP and MERT stop leading MGPHot once unofficial videos are excluded, the text-only matching is driving the result. A complementary check is to audio-fingerprint a sample of the unofficial uploads against the original recordings to measure how many are genuine matches.","tokens_in":7809,"feed_emoji":"🎵","tokens_out":14261,"duration_ms":126231,"temperature":0.7,"pith_summary":"Music autotagging — automatically assigning descriptive labels such as genre, mood, and instrumentation to audio — is the standard test bed for judging general-purpose music representations, and the paper argues that the datasets used for that test give a partial picture. To show this, the authors turn the expert-annotated MGPHot dataset — 21,320 Billboard Hot 100 tracks scored by professional musicians on 58 continuous musicological attributes spanning rhythm, harmony, instrumentation, sonority, vocals, and lyrics — into a working audio benchmark by matching every track to YouTube audio, defining an artist-disjoint train/val/test split, and releasing precomputed embeddings for seven state-of-the-art encoders. Across four tasks (three multilabel-tagging benchmarks and one regression) the paper finds that no single encoder leads in all settings: the genre-supervised MAEST tops the generic tag datasets, while CLAP, MERT, and even the speech-trained WHISPER share the top places on the expert annotations. If the result holds, claims that one model is state of the art in music understanding must be qualified by which labels were used for evaluation, and generic crowdsourced tag datasets alone cannot be trusted to rank representation quality.","feed_headline":"Expert music labels flip the autotagging leaderboard","feed_subtitle":"Seven encoders rank differently on 21,320 expert-annotated Billboard tracks: best model depends on the tag set.","key_machinery":"The load-bearing artifact is the MGPHot-audio benchmark extension: 21,320 Billboard Hot 100 tracks with expert musicological annotations — 58 continuous descriptors in seven categories (rhythm, composition, harmony, instrumentation, sonority, vocals, lyrics) — made usable for audio evaluation via a curated YouTube archive (56.43% official uploads), a canonical train/val/test split that is artist-disjoint and stratified by the 58 descriptors, release year, and official-uploader ratio, and precomputed embeddings for seven encoders. Two task formulations carry the evaluation: MGPHot-reg, regression on the continuous values scored by RMSE, and MGPHot-tag, multilabel classification on 174 binariz","core_discovery":"On the authors' terms, the discovery is that autotagging performance is benchmark-specific: rankings of seven pretrained audio encoders shift substantially when moving from generic crowdsourced tag datasets (MagnaTagATune, MTG-Jamendo) to MGPHot's expert musicological annotations. MAEST, pretrained on Discogs genre metadata, wins both generic datasets and is strongest in genre-related categories, whereas on MGPHot it drops below par. CLAP, trained with audio-text contrastive alignment, achieves the best MGPHot-tag score and is statistically tied with MERT for the best MGPHot-reg error. MERT and OMAR-RQ, self-supervised masked audio token prediction models, are consistently competitive everyw","pith_inferences":["WHISPER's top-3 showing on expert-annotated vocals and lyrics turns a speech model into a diagnostic tool: MGPHot-style expert labels could be used to decompose what different pretraining objectives actually capture acoustically, rather than only to rank whole models.","Because 43.57% of the archive's uploads are unofficial and matched by text alone, the benchmark's claims would be hardened by an audio-fingerprint audit and by confirming the reported rankings hold on official uploads only.","The year-stratified split (1958-2022) sets up a temporal-drift experiment the paper does not run: whether encoders trained on recent recordings lose accuracy on older music, and whether any model resists that drift.","The per-category and per-tag results suggest evaluating representations per musical attribute (vocals, harmony, production) instead of per dataset, since same-named categories behave differently across datasets."],"forward_implications":["No single frozen encoder is a reliable default for music autotagging; the best choice depends on the dataset and tag category, so 'state of the art' claims need cross-dataset qualification.","Conclusions about representation quality drawn from generic tag datasets alone (MagnaTagATune, MTG-Jamendo) should not be assumed to transfer to fine-grained, expert-level musical attributes.","Supervised genre pretraining (MAEST) pays off when downstream vocabulary matches genre-heavy annotations but loses ground when the task moves to non-genre musical descriptors; text-audio contrastive pretraining (CLAP) absorbs that mismatch better.","Self-supervised masked audio token prediction (MERT, OMAR-RQ) yields the most balanced performance across all settings without metadata supervision.","The released embeddings, splits, and reconstruction scripts let future models be probed under the same protocol without re-downloading audio or recomputing features, making the comparison directly reproducible."],"supporting_citations":[{"why":"Supplies the 21,320 expert-annotated tracks (58 musicological attributes per track) whose labels define the new benchmark and drive the comparison with generic tags.","marker":"[13]"},{"why":"Reference generic tag dataset; MAEST's lead here anchors the claim that rankings differ across datasets.","marker":"[11]"},{"why":"Reference generic tag dataset with a genre-heavy vocabulary that aligns with MAEST's pretraining, explaining its win and the observed ranking shift.","marker":"[12]"},{"why":"The genre-supervised encoder whose generic-dataset wins and MGPHot drop demonstrate vocabulary-alignment effects.","marker":"[16]"},{"why":"The audio-text contrastive encoder that achieves the best MGPHot-tag score and second-best MGPHot-reg error, backing the claim that it handles label mismatch better.","marker":"[15]"},{"why":"The masked-token self-supervised encoder statistically tied for the best MGPHot-reg error and consistently top-ranked in all four settings.","marker":"[17]"},{"why":"The speech-recognition encoder whose top-3 MGPHot result, driven by vocals and lyrics categories, is the paper's clearest ranking-shift evidence.","marker":"[14]"},{"why":"The multilabel masked-token encoder that stays competitive across all settings, supporting the balanced-performance conclusion for audio self-supervision.","marker":"[19]"},{"why":"The LLM used in the match-retrieval pipeline to resolve the 26.33% of tracks the title/artist regex missed; load-bearing for building the MGPHot audio archive.","marker":"[20]"}],"fun_headline_variants":["Autotagging rank flips when experts pick the labels","Which audio AI wins? Depends on who labels the songs","Expert tags reshuffle audio AI rankings","Best autotagging model? It depends on the tag source","Crowd tags vs expert tags: audio AI rankings flip"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The benchmark assumes the YouTube video retrieved for each track by title/artist matching — regex, then an LLM — is the actual recording the MGPHot annotations describe; 43.57% of matches are unofficial uploads and none were verified by audio content, so some labels may sit on different versions, covers, or unrelated audio.","fun_headline_variants_meta":{"raw":{"variants":["Autotagging rank flips when experts pick the labels","Which audio AI wins? Depends on who labels the songs","Expert tags reshuffle audio AI rankings","Best autotagging model? It depends on the tag source","Crowd tags vs expert tags: audio AI rankings flip"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000734,"raw_usage":{"total_tokens":3121,"prompt_tokens":746,"completion_tokens":2375,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":490,"completion_tokens_details":{"reasoning_tokens":2293}},"tokens_in":490,"tokens_out":2375,"duration_ms":17087,"temperature":1.0,"reasoning_tokens":2293,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T22:49:59.833067+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the paper's probe protocol on only the 56.43% of MGPHot tracks that come from official uploads and re-rank the seven encoders. The central claim — that rankings shift between generic and expert-annotated benchmarks — should survive on official-only audio; if CLAP and MERT stop leading MGPHot once unofficial videos are excluded, the text-only matching is driving the result. A complementary check is to audio-fingerprint a sample of the unofficial uploads against the original recordings to measure how many are genuine matches.","supporting_citations":[{"cited_title":"Supervised and unsupervised learning of audio representations for music understanding,","cited_arxiv_id":null,"evidence_quote":"Reference generic tag dataset; MAEST's lead here anchors the claim that rankings differ across datasets."},{"cited_title":"Cross- collection evaluation for music classification tasks,","cited_arxiv_id":null,"evidence_quote":"The audio-text contrastive encoder that achieves the best MGPHot-tag score and second-best MGPHot-reg error, backing the claim that it handles label mismatch better."},{"cited_title":"Faults in the latin music database and with its use,","cited_arxiv_id":null,"evidence_quote":"The masked-token self-supervised encoder statistically tied for the best MGPHot-reg error and consistently top-ranked in all four settings."},{"cited_title":"The latin music database,","cited_arxiv_id":null,"evidence_quote":"The speech-recognition encoder whose top-3 MGPHot result, driven by vocals and lyrics categories, is the paper's clearest ranking-shift evidence."},{"cited_title":"The mtg-jamendo dataset for automatic mu- sic tagging,","cited_arxiv_id":null,"evidence_quote":"The multilabel masked-token encoder that stays competitive across all settings, supporting the balanced-performance conclusion for audio self-supervision."},{"cited_title":"Mgphot: A dataset of musicological anno- tations for popular music (1958–2022),","cited_arxiv_id":null,"evidence_quote":"The LLM used in the match-retrieval pipeline to resolve the 26.33% of tracks the title/artist regex missed; load-bearing for building the MGPHot audio archive."}],"review_version":1}