Pith. sign in

REVIEW 2 major objections 5 minor 28 references

Benchmarking Music Autotagging with MGPHot Expert Annotations vs. Generic Tag Datasets

T0 review · 2 major / 5 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Turning expert-annotated MGPHot (21,320 Billboard tracks) into an audio benchmark, the paper shows no single encoder leads everywhere: rankings shift between generic tags and expert musical attributes.

desk verdict A genuinely useful benchmark resource for MIR that is currently held back by an unverified audio-identity assumption the authors need to address. read the letter →

arxiv 2509.06936 v1 pith:EHVK2I65 submitted 2025-09-08 cs.SD eess.AS

Pedro Ramoneda , Pablo Alonso-Jiménez , Sergio Oramas , Xavier Serra , Dmitry Bogdanov This is my paper
classification cs.SDeess.AS
keywords musicautotaggingMGPHotexpertmusicologicalannotationsaudiorepresentationlearningbenchmarkevaluationinformationretrievalBillboardHot100
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Music autotagging — automatically assigning descriptive labels such as genre, mood, and instrumentation to audio — is the standard test bed for judging general-purpose music representations, and the paper argues that the datasets used for that test give a partial picture. To show this, the authors turn the expert-annotated MGPHot dataset — 21,320 Billboard Hot 100 tracks scored by professional musicians on 58 continuous musicological attributes spanning rhythm, harmony, instrumentation, sonority, vocals, and lyrics — into a working audio benchmark by matching every track to YouTube audio, defining an artist-disjoint train/val/test split, and releasing precomputed embeddings for seven state-of-the-art encoders. Across four tasks (three multilabel-tagging benchmarks and one regression) the paper finds that no single encoder leads in all settings: the genre-supervised MAEST tops the generic tag datasets, while CLAP, MERT, and even the speech-trained WHISPER share the top places on the expert annotations. If the result holds, claims that one model is state of the art in music understanding must be qualified by which labels were used for evaluation, and generic crowdsourced tag datasets alone cannot be trusted to rank representation quality.

What carries the argument

The load-bearing artifact is the MGPHot-audio benchmark extension: 21,320 Billboard Hot 100 tracks with expert musicological annotations — 58 continuous descriptors in seven categories (rhythm, composition, harmony, instrumentation, sonority, vocals, lyrics) — made usable for audio evaluation via a curated YouTube archive (56.43% official uploads), a canonical train/val/test split that is artist-disjoint and stratified by the 58 descriptors, release year, and official-uploader ratio, and precomputed embeddings for seven encoders. Two task formulations carry the evaluation: MGPHot-reg, regression on the continuous values scored by RMSE, and MGPHot-tag, multilabel classification on 174 binariz

What would settle it

Re-run the paper's probe protocol on only the 56.43% of MGPHot tracks that come from official uploads and re-rank the seven encoders. The central claim — that rankings shift between generic and expert-annotated benchmarks — should survive on official-only audio; if CLAP and MERT stop leading MGPHot once unofficial videos are excluded, the text-only matching is driving the result. A complementary check is to audio-fingerprint a sample of the unofficial uploads against the original recordings to measure how many are genuine matches.

Watch

Extended reading notes

Core claim

On the authors' terms, the discovery is that autotagging performance is benchmark-specific: rankings of seven pretrained audio encoders shift substantially when moving from generic crowdsourced tag datasets (MagnaTagATune, MTG-Jamendo) to MGPHot's expert musicological annotations. MAEST, pretrained on Discogs genre metadata, wins both generic datasets and is strongest in genre-related categories, whereas on MGPHot it drops below par. CLAP, trained with audio-text contrastive alignment, achieves the best MGPHot-tag score and is statistically tied with MERT for the best MGPHot-reg error. MERT and OMAR-RQ, self-supervised masked audio token prediction models, are consistently competitive everyw

Load-bearing premise

The benchmark assumes the YouTube video retrieved for each track by title/artist matching — regex, then an LLM — is the actual recording the MGPHot annotations describe; 43.57% of matches are unofficial uploads and none were verified by audio content, so some labels may sit on different versions, covers, or unrelated audio.

Editorial extensions

If this is right

  • No single frozen encoder is a reliable default for music autotagging; the best choice depends on the dataset and tag category, so 'state of the art' claims need cross-dataset qualification.
  • Conclusions about representation quality drawn from generic tag datasets alone (MagnaTagATune, MTG-Jamendo) should not be assumed to transfer to fine-grained, expert-level musical attributes.
  • Supervised genre pretraining (MAEST) pays off when downstream vocabulary matches genre-heavy annotations but loses ground when the task moves to non-genre musical descriptors; text-audio contrastive pretraining (CLAP) absorbs that mismatch better.
  • Self-supervised masked audio token prediction (MERT, OMAR-RQ) yields the most balanced performance across all settings without metadata supervision.
  • The released embeddings, splits, and reconstruction scripts let future models be probed under the same protocol without re-downloading audio or recomputing features, making the comparison directly reproducible.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • WHISPER's top-3 showing on expert-annotated vocals and lyrics turns a speech model into a diagnostic tool: MGPHot-style expert labels could be used to decompose what different pretraining objectives actually capture acoustically, rather than only to rank whole models.
  • Because 43.57% of the archive's uploads are unofficial and matched by text alone, the benchmark's claims would be hardened by an audio-fingerprint audit and by confirming the reported rankings hold on official uploads only.
  • The year-stratified split (1958-2022) sets up a temporal-drift experiment the paper does not run: whether encoders trained on recent recordings lose accuracy on older music, and whether any model resists that drift.
  • The per-category and per-tag results suggest evaluating representations per musical attribute (vocals, harmony, production) instead of per dataset, since same-named categories behave differently across datasets.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper proposes a new music autotagging benchmark built on the MGPHot dataset of expert musicological annotations. It links all 21,320 MGPHot tracks to YouTube audio using a regex/LLM matching pipeline, defines a canonical train/val/test split with artist-disjoint stratification, and releases precomputed embeddings for seven audio encoders. The authors evaluate these encoders on MGPHot (regression and binarized classification) and on two generic tag datasets (MTG-Jamendo, MagnaTagATune), reporting per-category and per-tag performance. The central finding is that no single encoder leads across all datasets and categories, and that rankings shift between expert-annotated and generic tag datasets.

Significance. If the audio-identity concern is resolved, this is a valuable contribution: it extends MGPHot into a reusable audio benchmark, provides a carefully constructed split, and releases code, metadata, and embeddings that support reproducible evaluation. The probe protocol (frozen encoder + MLP, five seeds, significance tests) is rigorous, and the per-category analysis offers a more granular view than previous autotagging benchmarks. The finding that model rankings differ across label types would be an important caution for the MIR community. The release of precomputed features and reconstruction scripts is a concrete reproducibility asset.

major comments (2)
  1. [Section 2] The audio identity of the YouTube links is not verified. The pipeline matches tracks by title/artist regex (72.91%) and LLM checks over titles/descriptions (22.86% + 3.47%), but never checks whether the audio content corresponds to the exact MGPHot recording. Since 43.57% of accepted links are unofficial uploads, they may contain covers, live performances, remasters, or unrelated audio. MGPHot annotations are recording-specific (e.g., 'Vocal Grittiness', 'Harmonic sophistication'), so wrong audio would directly invalidate the MGPHot results in Table 3 and Figure 3. The paper should provide an audio fingerprinting check or, at minimum, a manual audit of a sample and a sensitivity analysis restricted to official uploads. Without this, the central claim of ranking shifts between MGPHot and generic datasets is not established.
  2. [Section 5 / Figure 3] The mapping of MTG-Jamendo and MagnaTagATune tags into categories (Genre, Instrument, Mood/Theme, etc.) is not described. The text states that the vocabularies are 'mapped into higher-level musical categories,' but the mapping itself is not given, nor is it mentioned as released. This makes the per-category heatmaps in Figure 3 non-reproducible and potentially arbitrary if categories were chosen after seeing results. The authors should release the tag-to-category mapping and justify the assignment (e.g., by using existing taxonomic groupings). This is load-bearing for the per-category ranking shifts that support the paper's conclusions.
minor comments (5)
  1. [Section 2] The criterion for 'official' uploads is not defined. It would be helpful to state whether 'official' is determined by channel type (e.g., artist topic, verified label) and how this was verified.
  2. [Section 3] The split stratification description is ambiguous: 'computed over all label bins' could refer to the 174 binarized tags or to continuous value bins. Please clarify whether stratification is performed on the continuous descriptors or on the binarized tags, and define the binning used.
  3. [Figure 3] The color scale for MGPHot-reg uses RMSE (lower is better) while the left panels use MAP (higher is better). This is stated in the caption, but the distinct scales and directions should be made more visually explicit in the figure itself.
  4. [Table 3] The notation 'bold and underlined' is explained, but the table also uses gray background for top-3. Please ensure the caption fully disambiguates these markings, especially when a cell is bold but not underlined.
  5. [Section 4] Early stopping with patience 50 is reported, but the number of training epochs or the validation metric used for early stopping is not given. This is a minor reproducibility detail.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: benchmark evaluation is self-contained; self-citations are not load-bearing.

full rationale

The paper is an empirical benchmark study, not a derivation. The central claims—that no single encoder leads in all settings and that rankings shift across datasets—are obtained by evaluating seven pretrained encoders on three datasets with a fixed probing protocol. The MGPHot labels are externally sourced expert annotations from the Music Genome Project; the split is stratified by descriptor distributions, year, and official-uploader balance, but no target label is used to construct audio embeddings or to select the models. The use of OMAR-RQ and the MGPHot dataset involves self-citations, but these are not used as proof of the paper's findings: OMAR-RQ is evaluated on the same footing as the other encoders, and MGPHot's annotations were published independently with external provenance. The unverified audio identity of unofficial YouTube links is a validity threat, not a circularity: it concerns measurement error, not the reduction of an output to an input. No equation or fitted parameter is renamed as a prediction, and no uniqueness theorem or ansatz is imported from the authors' prior work to force the conclusion. Therefore the paper does not exhibit circular reasoning; at most it contains minor self-citations that are not load-bearing.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The benchmark has no fitted parameters in the model sense, but the tag discretization and split acceptance thresholds are hand-chosen. The load-bearing assumptions are external to the math: the audio-label correspondence and the reliability of the expert annotations.

free parameters (2)
  • MGPHot-tag bin boundaries = 0.33 and 0.66
    Hand-chosen cutoffs that discretize continuous [0,1] expert annotations into Low/Moderate/High tags; no principled derivation.
  • split stratification tolerance = 2%
    Hand-chosen maximum absolute difference in label proportions between each split and the full dataset; determines when a candidate split is accepted.
assumptions (3)
  • domain assumption YouTube audio matches the original annotated recording
    Section 2 uses title/artist matching only; no audio fingerprinting. If a video is a cover, remaster, or wrong track, the labels describe a different recording.
  • domain assumption MGPHot expert annotations are reliable ground truth
    The benchmark treats Music Genome Project annotations as correct without independent validation or inter-annotator agreement data.
  • domain assumption Frozen-encoder probing measures representation quality
    The paper acknowledges in Section 6 that full fine-tuning could raise performance; the protocol assumes the frozen representation is a fair basis for comparison.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Benchmarking Music Autotagging with MGPHot Expert Annotations vs. Generic Tag Datasets." pith.science (2026). https://pith.science/paper/EHVK2I65

@misc{pith2026250906936,
  author       = {Pith},
  title        = {Pith review of: Benchmarking Music Autotagging with MGPHot Expert Annotations vs. Generic Tag Datasets},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EHVK2I65}},
  note         = {Machine review of arXiv:2509.06936}
}
read the original abstract

Music autotagging aims to automatically assign descriptive tags, such as genre, mood, or instrumentation, to audio recordings. Due to its challenges, diversity of semantic descriptions, and practical value in various applications, it has become a common downstream task for evaluating the performance of general-purpose music representations learned from audio data. We introduce a new benchmarking dataset based on the recently published MGPHot dataset, which includes expert musicological annotations, allowing for additional insights and comparisons with results obtained on common generic tag datasets. While MGPHot annotations have been shown to be useful for computational musicology, the original dataset neither includes audio nor provides evaluation setups for its use as a standardized autotagging benchmark. To address this, we provide a curated set of YouTube URLs with retrievable audio, and propose a train/val/test split for standardized evaluation, and precomputed representations for seven state-of-the-art models. Using these resources, we evaluated these models in MGPHot and standard reference tag datasets, highlighting key differences between expert and generic tag annotations. Altogether, our contributions provide a more advanced benchmarking framework for future research in music understanding.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

28 extracted references · 25 canonical work pages

  1. [1]

    Benchmarking Music Autotagging with MGPHot Expert Annotations vs. Generic Tag Datasets

    INTRODUCTION Music autotagging aims to derive rich semantic descriptors, such as genre, mood, instrumentation, rhythm, harmony, production, and composition traits, directly from raw au- dio [1, 2, 3]. Such an analysis has great potential in various applications, especially in music streaming and recommen- dation services and in the management of music cat...

    work page Pith review arXiv 1958

  2. [2]

    Low” (0,0.33), “Moderate

    GA THERING AUDIO FORMGPHOT Figure 1 illustrates the pipeline we followed to collect YouTube URLs for the metadata ofMGPHot. We started from the meta- data for the 21,320 chart tracks. For each track, we searched YouTube using the title of the song and the artist’s name, keep- ing the top five results. A regular expression match between the track title and...

    work page

  3. [3]

    We start from the full collection

    MGPHOT DA TASET PARTITIONING Figure 2 sketches the automatic procedure used to create the canonical train/val/test split forMGPHot. We start from the full collection. For conducting the iterative split generation, each candidate split must satisfy four constraints: •Stratification by the 58 expert descriptors.We match the marginal distribution of every de...

    work page

  4. [4]

    ForMGPHot we use our proposed split

    EV ALUA TION PROTOCOL Dataset splits.We follow the train/validation/test partition used in previous work forMagnaTagATune[18, 19], and thesplit 0base autotagging partition forMTG–Jamendo. ForMGPHot we use our proposed split. Tasks.We consider two tagging settings. ForMagnaTa- gATune,MTG–Jamendo, andMGPHot-tag, we perform mul- tilabel classification with s...

    work page

  5. [5]

    Instrument

    RESULTS Table 3 reports the mean average precision (MAP↑) for the three tagging tasks and root mean-squared error (RMSE↓) for the regression task.4 Each score is the mean of five runs initial- ized with different seeds. 4Chosen for interpretability, MAE and MSE results are available online. Model Task Hoursθ(M) Architecture WHISPERASR 680,000 635 Transfor...

    work page

  6. [6]

    This distribution of win- ners indicates that there is no single reliable choice

    DISCUSSION AND LIMITA TIONS Although all encoders considered claim state-of-the-art perfor- mance, our study finds no model that consistently leads across all settings.MAESTachieves the best scores in the two generic tag datasets,CLAP,WHISPER, andMERTshare the top position in detailed musical features annotated by experts andOMAR- RQremains competitive in...

    work page

  7. [7]

    IA y M´usica: C ´atedra en Inteligen- cia Artificial y M ´usica

    CONCLUSION In this paper, we evaluate state-of-the-art music audio represen- tations in music autotagging tasks, using two common generic tag datasets and a newMGPHotdataset, which we extend and propose as a new benchmark for audio-based evaluations. The results reveal performance inconsistencies across datasets, highlighting the limitations of relying so...

    work page 2023

  8. [8]

    Automatic tagging of audio: The state-of-the-art,

    T. Bertin-Mahieux, D. Eck, and M. Mandel, “Automatic tagging of audio: The state-of-the-art,” inMachine audi- tion: Principles, algorithms and systems. IGI Global, 2011, pp. 334–352

    work page 2011

Show all 28 references
  1. [9]

    A survey of tagging techniques for music, speech and environmental sound,

    S. Duan, J. Zhang, P. Roe, and M. Towsey, “A survey of tagging techniques for music, speech and environmental sound,”Artificial Intelligence Review, vol. 42, no. 4, pp. 637–661, 2014

    2014

  2. [10]

    Three current issues in music autotagging,

    G. Marques, M. A. Domingues, T. Langlois, and F. Gouyon, “Three current issues in music autotagging,” inProceedings of the 12th International Society for Mu- sic Information Retrieval Conference (ISMIR), Miami, Florida, USA, 2011

    2011

  3. [11]

    Supervised and unsupervised learning of audio representations for music understanding,

    M. C. McCallum, F. Korzeniowski, S. Oramas, F. Gouyon, and A. F. Ehmann, “Supervised and unsupervised learning of audio representations for music understanding,” inPro- ceedings of the 23rd International Society for Music Infor- mation Retrieval Conference (ISMIR), Bengaluru, ...

    2022

  4. [12]

    Foundation models for music: A survey,

    Y . Ma, A. Øland, A. Ragni, B. M. Del Sette, C. Saitis, C. Donahue, C. Lin, C. Plachouras, E. Benetos, E. Shatri et al., “Foundation models for music: A survey,”arXiv preprint arXiv:2408.14340, 2024

    2024 arXiv

  5. [13]

    Musical genre classification of audio signals,

    G. Tzanetakis and P. Cook, “Musical genre classification of audio signals,”IEEE Transactions on speech and audio processing, vol. 10, no. 5, pp. 293–302, 2002

    2002

  6. [14]

    The latin music database,

    C. N. Silla Jr., A. L. Koerich, and C. A. A. Kaestner, “The latin music database,” inProceedings of the 9th Inter- national Conference on Music Information Retrieval (IS- MIR 2008). Philadelphia, PA, USA: International Society for Music Information Retrieval, 2008

    2008

  7. [15]

    Cross- collection evaluation for music classification tasks,

    D. Bogdanov, A. Porter, P. Herrera, and X. Serra, “Cross- collection evaluation for music classification tasks,” in Proceedings of the 17th International Society for Music Information Retrieval Conference (ISMIR), 2016

    2016

  8. [16]

    The gtzan dataset: Its contents, its faults, their effects on evaluation, and its future use,

    B. L. Sturm, “The gtzan dataset: Its contents, its faults, their effects on evaluation, and its future use,”arXiv preprint arXiv:1306.1461, 2013

    2013 arXiv

  9. [17]

    Faults in the latin music database and with its use,

    ——, “Faults in the latin music database and with its use,” inExtended Abstracts for the Late-Breaking Demo Ses- sion of the 16th International Society for Music Informa- tion Retrieval Conference (ISMIR), Oct. 2015

    2015

  10. [18]

    Evaluation of algorithms using games: The case of mu- sic tagging,

    E. Law, K. West, M. I. Mandel, M. Bay, and J. S. Downie, “Evaluation of algorithms using games: The case of mu- sic tagging,” inProc. 10th Int. Soc. Music Information Re- trieval Conf. (ISMIR), 2009

    2009

  11. [19]

    The mtg-jamendo dataset for automatic mu- sic tagging,

    D. Bogdanov, M. Won, P. Tovstogan, A. Porter, and X. Serra, “The mtg-jamendo dataset for automatic mu- sic tagging,” inMachine Learning for Music Discovery Workshop, International Conference on Machine Learn- ing (ICML), 2019

    2019

  12. [20]

    Mgphot: A dataset of musicological anno- tations for popular music (1958–2022),

    S. Oramas, F. Gouyon, S. Hogan, C. Landau, and A. Ehmann, “Mgphot: A dataset of musicological anno- tations for popular music (1958–2022),”Transactions of the International Society for Music Information Retrieval, vol. 8, no. 1, pp. 108–120, 2025

    1958

  13. [21]

    Robust speech recogni- tion via large-scale weak supervision,

    A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recogni- tion via large-scale weak supervision,” inProceedings of the 40th International Conference on Machine Learning, ser. Proceedings of Machine Learning Research. PMLR, July 2023

    2023

  14. [22]

    Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation,

    Y . Wu*, K. Chen*, T. Zhang*, Y . Hui*, T. Berg- Kirkpatrick, and S. Dubnov, “Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation,” inIEEE International Conference on Acoustics, Speech and Signal Processing, (ICASSP), 2023

    2023

  15. [23]

    Efficient supervised training of audio transformers for music rep- resentation learning,

    P. Alonso-Jim ´enez, X. Serra, and D. Bogdanov, “Efficient supervised training of audio transformers for music rep- resentation learning,” inProceedings of the 24th Interna- tional Society for Music Information Retrieval Conference (ISMIR), Milan, Italy, 2023

    2023

  16. [24]

    Mert: Acoustic music un- derstanding model with large-scale self-supervised train- ing,

    Y . Li, R. Yuan, G. Zhang, Y . Ma, X. Chen, H. Yin, C. Xiao, C. Lin, A. Ragni, E. Benetos, N. Gyenge, R. Dan- nenberg, R. Liu, W. Chen, G. Xia, Y . Shi, W. Huang, Z. Wang, Y . Guo, and J. Fu, “Mert: Acoustic music un- derstanding model with large-scale self-supervised train- i...

    2024

  17. [25]

    A foundation model for music informatics,

    M. Won, Y .-N. Hung, and D. Le, “A foundation model for music informatics,” inIEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2024

    2024

  18. [26]

    OMAR-RQ: Open music audio representation model trained with multi-feature masked token prediction,

    P. Alonso-Jim ´enez, P. Ramoneda, R. O. Araz, A. Poltron- ieri, and D. Bogdanov, “OMAR-RQ: Open music audio representation model trained with multi-feature masked token prediction,” inACM Multimedia Confer- ence (ACMMM), Open Source Track, 2025

    2025

  19. [27]

    Qwen2.5: A party of foundation models,

    Q. Team, “Qwen2.5: A party of foundation models,” September 2024. [Online]. Available: https://qwenlm. github.io/blog/qwen2.5/

    2024

  20. [28]

    Green mir?: Investigating computational cost of recent music- ai research in ismir,

    A. Holzapfel, A.-K. Kaila, and P. J ¨a¨askel¨ainen, “Green mir?: Investigating computational cost of recent music- ai research in ismir,” inInternational Society for Music Information Retrieval Conference (ISMIR), 2024

    2024

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.