Pith. sign in

REVIEW 3 major objections 5 minor 28 references

Benchmarking Music Generation Models and Metrics via Human Preference Studies

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A 15,600-comparison human study ranks music generation models: Suno v3.5 wins, and music-trained CLAP embeddings best match human judgment.

desk verdict A useful open benchmark for music generation evaluation, but the headline metric ranking rests on 13 points with no error bars. read the letter →

arxiv 2506.19085 v1 pith:XKE47UZC submitted 2025-06-23 cs.LG cs.SD

Florian Grötschla , Ahmet Solak , Luca A. Lanzendörfer , Roger Wattenhofer This is my paper · ORCID
classification cs.LGcs.SD
keywords musicgenerationhumanpreferencetext-audioalignmentFrechetAudioDistanceCLAPembeddingsEloratingbenchmarkdataset
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks which music-generation models people actually prefer and which automated metrics come closest to matching those preferences. To answer, the authors generated 6,000 songs with 12 current models, ran 15,600 pairwise listening comparisons with more than 2,500 participants, and ranked models by bootstrapped Elo ratings and Bradley-Terry strength. They find that the commercial model Suno v3.5 is preferred over all others and also matches text prompts best, beating even the human-made reference dataset. On the metric side, CLAP embedding models trained on music data—both inside Frechet Audio Distance for quality and as cosine-similarity scorers for text alignment—correlate most strongly with the human judgments. If correct, music-trained CLAP embeddings are the best cheap proxy for human taste in generated music, and the released dataset gives the field a fixed benchmark for testing new metrics.

What carries the argument

The engine is pairwise binary preference testing on 10-second instrumental clips, with the same tag triple used to generate both clips so that preference and text-audio alignment are judged on matched content. Human choices are converted into model strengths via bootstrapped Elo ratings and Bradley-Terry parameters. Those human strengths are then correlated, by Pearson and Spearman coefficients, against objective scores: Frechet Audio Distance computed on VGGish, PANN, EnCodec, and several CLAP embedding spaces for music quality, and CLAP cosine similarity between audio and prompt text for text-audio alignment. The decisive comparison is which embedding space makes FAD or cosine similarity track the human rank ordering.

What would settle it

Re-run the same pairwise comparison survey with full-length vocal tracks and a listener pool that includes older adults and non-streamers, then check whether Suno v3.5 still leads and whether FAD-CLAP-MA and the music-trained CLAP alignment scores still correlate with human ratings; if the ranking shifts or the correlations drop materially, the 10-second instrumental-clip proxy is the load-bearing choice.

Watch

Extended reading notes

Core claim

The central discovery is an empirical ranking grounded in human pairwise choices rather than in any single automated score. Among 12 generation models, Suno v3.5 earns the highest bootstrapped Elo in both music preference and text-audio alignment, with Suno v3 and Udio also surpassing the reference corpus. On the metric side, FAD computed with the music-audioset CLAP checkpoint (FAD-CLAP-MA) has the best Pearson and Spearman correlation with human music-preference Bradley-Terry parameters, and the music-trained CLAP checkpoints give the highest correlation with human text-audio alignment judgments. The authors read this as evidence that CLAP models trained on music data approximate human preferences most accurately, both as embedding models for FAD and for measuring text-audio alignment.

Load-bearing premise

The results assume that binary preference between 10-second instrumental clips, judged by fluent English speakers aged 18 to 34 who use streaming services, captures what music quality and text-audio alignment mean; if full-length structure, vocals, or other listener groups matter, both the model ranking and the metric correlations could change.

Editorial extensions

If this is right

  • Suno v3.5 becomes the baseline to beat for human-preferred text-to-music generation among currently available models.
  • FAD-CLAP-MA can substitute for expensive listening tests when comparing music generation models on quality.
  • Music-trained CLAP cosine similarity can automate text-audio alignment checks during data filtering or model development.
  • New metrics can be validated against the released human preference data without running a new 2,500-participant study.
  • Within a model family, the newer and larger checkpoint generally outranked the older or smaller one, pointing to scale and iteration as reliable improvement levers.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper only tests 10-second instrumental clips, so its ranking is an inference about short-form instrumental snippets; judging 30-to-60-second clips with vocals could change the model order, especially on structural coherence and vocal quality.
  • The listener pool is fluent English speakers aged 18 to 34 who use streaming services; broader age and cultural groups could rank models differently, particularly on genre fit.
  • If music-trained CLAP embeddings really track human preference this well, they could serve as a training-time reward signal or a filter for selecting generated samples, not just a post-hoc evaluation metric.
  • The released dataset lets future metric developers test against a fixed human ground truth; the next step is checking whether any new embedding improves on FAD-CLAP-MA at the clip level, not just at the model-aggregate level.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper presents a large-scale benchmark of 12 music generation models using 6,000 generated tracks and 15,600 pairwise human preference comparisons collected from over 2,500 participants via Prolific. Prompts are selected from MTG-Jamendo tag combinations filtered for diversity with a CLAP-based cosine-similarity threshold. The authors compute Elo ratings and Bradley-Terry parameters from the human comparisons, then correlate objective metrics (FAD variants and CLAP/LAION text-audio alignment scores) with these human-derived strengths across 13 model-level data points. They conclude that Suno v3.5 is the most preferred and best-aligned model and that music-trained CLAP embeddings (FAD-CLAP-MA and LAION-MA) best approximate human preferences. All generated audio, prompts, and human response data are released openly.

Significance. If the conclusions are statistically robust, this would be one of the first large-scale public resources for comparing music generation models and objective metrics against human preference, and the dataset release is a valuable contribution to the community. The use of Elo and Bradley-Terry models is standard, and the bootstrapping of Elo ratings is a strength, as is the broad coverage of commercial and open-source models. However, the central metric-ranking claim rests on a small set of model-level correlation coefficients without uncertainty quantification, and the prompt-selection procedure may advantage CLAP-family metrics. These issues currently prevent the paper's headline conclusions from being fully established.

major comments (3)
  1. [Section V, Fig. 5] The central claim that music-trained CLAP embeddings best approximate human preference rests on Pearson and Spearman correlations computed over only 13 model-level data points (12 models plus MTG-Jamendo). With n=13, rank correlations have very wide sampling distributions and are highly sensitive to a single observation. The paper reports no confidence intervals, significance tests, permutation tests, or leave-one-out robustness checks. Please add bootstrap confidence intervals that resample both human comparisons and models, and report leave-one-model-out correlations to show that the ordering in Fig. 5 is not driven by one model. Without this, the metric ranking and the Section VI conclusion are not statistically established.
  2. [Section III-A] The prompt set is selected by requiring that no two tag combinations have CLAP embedding cosine similarity above a threshold of 0.1382. The metrics evaluated later include the same CLAP family (FAD-CLAP-MA, LAION-MA, and other LAION checkpoints). This creates a selection bias: the test distribution is constructed to be well-separated in the CLAP embedding space, potentially inflating the measured performance of CLAP-based metrics relative to metrics based on other embeddings. Please test robustness by selecting prompts with an alternative diversity criterion (e.g., random selection or diversity under VGGish/PANN embeddings) and check whether the metric ranking in Fig. 5 is preserved.
  3. [Sections III-B and IV-A] The human evaluation operationalizes 'music quality' and 'text-audio alignment' through 10-second instrumental excerpts judged by fluent English speakers aged 18 to 34 who use streaming services. The abstract and conclusions state the results in terms of 'human preference' without these qualifiers. Since full-song structure, vocals, and listener demographics may change both model ranking and metric correlations, the claims should be scoped to this experimental setup, or the paper should provide evidence that the 10-second instrumental proxy is representative of full-length listening. Please add a limitations paragraph and soften the corresponding claims.
minor comments (5)
  1. [References] Reference [21] spells the platform name as 'Prolofic'; it should be 'Prolific'.
  2. [Section IV-B] The statement that 'Suno v3.5 achieved a significantly higher Elo than all other models' needs a significance test or confidence intervals; the current text reports only the mean over bootstraps.
  3. [Figures 3 and 4] The color scale and cell values are not explained in the captions; please add a legend and define whether green always indicates 'better according to the metric' or 'better according to human evaluation'.
  4. [Abstract and Section I] The numbers 15k and 6k are used loosely; the abstract later states 15,600 comparisons and 6,000 songs. Please be consistent and use exact numbers throughout.
  5. [Section VI] The phrase 'providing an unbiased ranking of the models' is too strong given the prompt-selection procedure and the participant filter; please replace 'unbiased' with a more cautious formulation.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: human preference judgments provide an external benchmark, and no metric is fitted to the human ratings.

full rationale

The paper's central claims are empirical comparisons between objective metrics and human preference judgments. The human pairwise comparisons (Bradley-Terry parameters) are collected independently of the metrics, and no metric parameter is fitted to those judgments. The use of CLAP embeddings to select diverse tag combinations (Section III-A) is a dataset-design choice; it does not constrain the human judgments or the computed metric scores to match each other, so it does not make the CLAP metric ranking circular. No load-bearing self-citation or imported uniqueness theorem appears; references to prior FAD work are independent external results. The statistical fragility of the 13-point correlation analysis (no confidence intervals, no leave-one-out) is a robustness concern, not a circularity concern. Accordingly, the derivation chain is self-contained against an external benchmark.

Assumptions & free parameters 4 free parameters · 8 assumptions · 0 invented entities

The benchmark introduces no invented physical or model entities. It rests on domain assumptions about what human preference measures and about the validity of embedding-based metrics; its only hand-fitted design parameters are the prompt diversity threshold, survey sample size, clip duration, and Elo hyperparameters.

free parameters (4)
  • CLAP tag-diversity threshold = 0.1382
    Selected by binary search so that exactly 500 tag combinations remain; controls prompt diversity and all downstream generation and evaluation.
  • Elo K-factor = 8
    A standard choice for Elo updates; affects rating magnitudes and the bootstrap variance but not the reported ranking order.
  • Survey tag-combination sample size = 100
    Randomly selected from the 500 prompt combinations; the model rankings depend on this sample of prompts and the corresponding generated tracks.
  • Clip duration and segment selection = 10 seconds, highest-energy segment
    All clips are truncated to 10-second instrumental versions; longer tracks are reduced to the highest-energy segment. This design choice affects every model score and preference judgment.
assumptions (8)
  • domain assumption Human pairwise preference is the gold standard for music quality and text-audio alignment.
    Section I states that human preference is the gold standard, and the entire benchmark is built on this premise without justifying pairwise binary preference as the correct operationalization.
  • domain assumption Pairwise comparisons of 10-second instrumental clips capture the qualities that matter for full music generation.
    Sections III-B and IV-A restrict all audio to 10-second instrumental excerpts; the paper does not test whether rankings generalize to full songs or vocal music.
  • domain assumption MTG-Jamendo tags are representative text descriptions for evaluating text-audio alignment.
    Section III uses MTG-Jamendo tags, filtered by presence in FMA and MagnaTagATune, as prompts; the prompt distribution drives all model scores.
  • domain assumption The Prolific participant pool represents human listeners broadly enough for a general benchmark.
    Section IV-A restricts participants to fluent English speakers aged 18 to 34 who use music streaming services; no evidence is given that preferences are stable across age, language, or listening habits.
  • domain assumption FAD computed on VGGish, PANN, CLAP, or EnCodec embeddings is a valid perceptual quality metric.
    Section V uses FAD variants as proxies for music quality; this is part of the premise being tested rather than an independently established fact.
  • domain assumption CLAP cosine similarity is a valid proxy for text-audio alignment.
    Section V computes mean cosine similarity between audio and tag text embeddings; the conclusion that these correlate with human alignment depends on this embedding space capturing alignment-relevant information.
  • ad hoc to paper CLAP-based prompt selection does not bias the subsequent metric evaluation.
    Section III-A selects diverse prompts using CLAP embeddings with threshold 0.1382; since the same model family is later ranked for alignment, this could inflate CLAP metrics' apparent agreement with human judgments.
  • standard math Elo rating updates and Bradley-Terry model inference recover latent model strengths from pairwise comparisons.
    Sections IV-B and V rely on these standard ranking models without proof; they are accepted background in the survey and ranking literature.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Benchmarking Music Generation Models and Metrics via Human Preference Studies." pith.science (2026). https://pith.science/paper/XKE47UZC

@misc{pith2026250619085,
  author       = {Pith},
  title        = {Pith review of: Benchmarking Music Generation Models and Metrics via Human Preference Studies},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XKE47UZC}},
  note         = {Machine review of arXiv:2506.19085}
}
read the original abstract

Recent advancements have brought generated music closer to human-created compositions, yet evaluating these models remains challenging. While human preference is the gold standard for assessing quality, translating these subjective judgments into objective metrics, particularly for text-audio alignment and music quality, has proven difficult. In this work, we generate 6k songs using 12 state-of-the-art models and conduct a survey of 15k pairwise audio comparisons with 2.5k human participants to evaluate the correlation between human preferences and widely used metrics. To the best of our knowledge, this work is the first to rank current state-of-the-art music generation models and metrics based on human preference. To further the field of subjective metric evaluation, we provide open access to our dataset of generated music and human evaluations.

Figures

Figures reproduced from arXiv: 2506.19085 by the authors.

Figure 1
Figure 1. Elo ratings for all music generation models in the music preference [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of the tag selection, music generation, and study. We extract common and diverse tag combinations from MTG-Jamendo and use them to [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Scores of the tested metrics for music preference estimation on [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (1 more)
Figure 5
Figure 5. Figure 5: Pearson correlation coefficient (PCC) and Spearman’s rank correlation [PITH_FULL_IMAGE:figures/full_fig_p004_5.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

28 extracted references · 11 canonical work pages

  1. [1]

    Fr\’echet audio distance: A metric for evaluating music enhancement algorithms,

    K. Kilgour, M. Zuluaga, D. Roblek, and M. Sharifi, “Fr\’echet audio distance: A metric for evaluating music enhancement algorithms,”arXiv preprint arXiv:1812.08466, 2018

    arXiv 2018

  2. [2]

    Cnn architectures for large-scale audio classification,

    S. Hershey, S. Chaudhuri, D. P. Ellis, J. F. Gemmeke, A. Jansen, R. C. Moore, M. Plakal, D. Platt, R. A. Saurous, B. Seyboldet al., “Cnn architectures for large-scale audio classification,” in2017 ieee international conference on acoustics, speech and signal processing (icassp). IEEE, 2017, pp. 131–135

    2017

  3. [3]

    Musiclm: Generating music from text,

    A. Agostinelli, T. I. Denk, Z. Borsos, J. Engel, M. Verzetti, A. Caillon, Q. Huang, A. Jansen, A. Roberts, M. Tagliasacchiet al., “Musiclm: Generating music from text,”arXiv preprint arXiv:2301.11325, 2023

    arXiv 2023

  4. [4]

    Riffusion-stable diffusion for real-time music generation,

    S. Forsgren and H. Martiros, “Riffusion-stable diffusion for real-time music generation,”URL https://riffusion. com, 2022

    2022

  5. [5]

    Clap learning audio concepts from natural language supervision,

    B. Elizalde, S. Deshmukh, M. Al Ismail, and H. Wang, “Clap learning audio concepts from natural language supervision,” inICASSP 2023- 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023, pp. 1–5

    2023

  6. [6]

    Natural language supervision for general-purpose audio representations,

    B. Elizalde, S. Deshmukh, and H. Wang, “Natural language supervision for general-purpose audio representations,” inICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024, pp. 336–340

    work page 2024

  7. [7]

    Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation,

    Y . Wu, K. Chen, T. Zhang, Y . Hui, T. Berg-Kirkpatrick, and S. Dubnov, “Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation,” inICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023, pp. 1–5

    2023

  8. [8]

    Evaluating generative audio systems and their metrics

    A. Vinay and A. Lerch, “Evaluating generative audio systems and their metrics,”arXiv preprint arXiv:2209.00130, 2022

    work page Pith review arXiv 2022

Show all 28 references
  1. [9]

    Adapting frechet audio distance for generative music evaluation,

    A. Gui, H. Gamper, S. Braun, and D. Emmanouilidou, “Adapting frechet audio distance for generative music evaluation,” inICASSP 2024- 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024, pp. 1331–1335

    2024

  2. [10]

    Noise2music: Text-conditioned music generation with diffusion models,

    Q. Huang, D. S. Park, T. Wang, T. I. Denk, A. Ly, N. Chen, Z. Zhang, Z. Zhang, J. Yu, C. Franket al., “Noise2music: Text-conditioned music generation with diffusion models,”arXiv preprint arXiv:2302.03917, 2023

    2023 arXiv

  3. [11]

    The mtg-jamendo dataset for automatic music tagging,

    D. Bogdanov, M. Won, P. Tovstogan, A. Porter, and X. Serra, “The mtg-jamendo dataset for automatic music tagging,” inMachine Learning for Music Discovery Workshop, International Conference on Machine Learning (ICML 2019), Long Beach, CA, United States,

    2019

  4. [12]

    Fma: A dataset for music analysis,

    M. Defferrard, K. Benzi, P. Vandergheynst, and X. Bresson, “Fma: A dataset for music analysis,”arXiv preprint arXiv:1612.01840, 2016

    2016 arXiv

  5. [13]

    Evaluation of algorithms using games: The case of music tagging

    E. Law, K. West, M. I. Mandel, M. Bay, and J. S. Downie, “Evaluation of algorithms using games: The case of music tagging.” inISMIR. Citeseer, 2009, pp. 387–392

    2009

  6. [14]

    Simple and controllable music generation,

    J. Copet, F. Kreuk, I. Gat, T. Remez, D. Kant, G. Synnaeve, Y . Adi, and A. D ´efossez, “Simple and controllable music generation,”Advances in Neural Information Processing Systems, vol. 36, 2024

    2024

  7. [15]

    ”audioldm 2: Learning holis- tic audio generation with self-supervised pretraining

    H. Liu, Q. Tian, Y . Yuan, X. Liu, X. Mei, Q. Kong, Y . Wang, W. Wang, Y . Wang, and M. D. Plumbley, “”audioldm 2: Learning holis- tic audio generation with self-supervised pretraining”,”arXiv preprint arXiv:2308.05734, 2023

    2023 arXiv

  8. [16]

    Mustango: Toward controllable text-to-music generation,

    J. Melechovsky, Z. Guo, D. Ghosal, N. Majumder, D. Herremans, and S. Poria, “Mustango: Toward controllable text-to-music generation,” arXiv preprint arXiv:2311.08355, 2023

    2023 arXiv

  9. [17]

    Fast timing- conditioned latent audio diffusion,

    Z. Evans, C. Carr, J. Taylor, S. H. Hawley, and J. Pons, “Fast timing- conditioned latent audio diffusion,”arXiv preprint arXiv:2402.04825, 2024

    2024 arXiv

  10. [18]

    Suno, “Suno,” https://suno.com/, n.d., accessed: May-June, 2024

    2024

  11. [19]

    Udio, “Udio,” https://www.udio.com/, n.d., accessed: May-July, 2024

    2024

  12. [20]

    Chatbot arena: An open platform for evaluating llms by human preference,

    W.-L. Chiang, L. Zheng, Y . Sheng, A. N. Angelopoulos, T. Li, D. Li, H. Zhang, B. Zhu, M. Jordan, J. E. Gonzalezet al., “Chatbot arena: An open platform for evaluating llms by human preference,”arXiv preprint arXiv:2403.04132, 2024

    2024 arXiv

  13. [21]

    Prolific,

    Prolofic, “Prolific,” https://www.prolific.com/, n.d., accessed: 2024

    2024

  14. [22]

    Amazon mechanical turk,

    A. M. Turk, “Amazon mechanical turk,” https://www.mturk.com/, n.d., accessed: 2024

    2024

  15. [23]

    Data quality in on- line human-subjects research: Comparisons between mturk, prolific, cloudresearch, qualtrics, and sona,

    B. D. Douglas, P. J. Ewell, and M. Brauer, “Data quality in on- line human-subjects research: Comparisons between mturk, prolific, cloudresearch, qualtrics, and sona,”Plos one, vol. 18, no. 3, p. e0279720, 2023

    2023

  16. [24]

    Fast and accurate inference of plackett–luce models,

    L. Maystre and M. Grossglauser, “Fast and accurate inference of plackett–luce models,”Advances in neural information processing sys- tems, vol. 28, 2015

    2015

  17. [25]

    Rank analysis of incomplete block designs: I. the method of paired comparisons,

    R. A. Bradley and M. E. Terry, “Rank analysis of incomplete block designs: I. the method of paired comparisons,”Biometrika, vol. 39, no. 3/4, pp. 324–345, 1952

    1952

  18. [26]

    Panns: Large-scale pretrained audio neural networks for audio pattern recognition,

    Q. Kong, Y . Cao, T. Iqbal, Y . Wang, W. Wang, and M. D. Plumbley, “Panns: Large-scale pretrained audio neural networks for audio pattern recognition,”IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 28, pp. 2880–2894, 2020

    2020

  19. [27]

    High fidelity neural audio compression,

    A. D ´efossez, J. Copet, G. Synnaeve, and Y . Adi, “High fidelity neural audio compression,”arXiv preprint arXiv:2210.13438, 2022

    2022 arXiv

  20. [2019]

    Available: http://hdl.handle.net/10230/42015

    [Online]. Available: http://hdl.handle.net/10230/42015

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.