Pith. sign in

REVIEW 2 major objections 4 minor 18 references

Audio models fail at music aesthetics; text comments carry the signal

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · glm-5.2

2026-07-09 22:43 UTC pith:LX23PVBG

load-bearing objection Solid dataset contribution with real annotation rigor; methodological gaps are second-order and don't undermine the core finding. the 2 major comments →

arxiv 2607.06929 v1 pith:LX23PVBG submitted 2026-07-08 cs.SD cs.AI

MADB: A Large-Scale Music Aesthetics Dataset with Professional and Multi-Dimensional Annotations

Sirui Zhang , Tianle Wang , Xinyi Tong , Peiyang Yu , Jishang Chen , Liangke Zhao , Haoxin Zhang , Duo Xu
show 3 more authors
Xin Jin Feng Yu Songchun Zhu
This is my paper
classification cs.SD cs.AI
keywords music aesthetic assessmentmulti-dimensional annotationaudio representation learningmultimodal evaluationbenchmark datasethuman-aligned perceptioncontrastive learning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces a dataset of 9,999 music tracks rated by 30 professionally trained annotators across 10 perceptual dimensions (such as melody perception, arrangement emotion, and singing skill) plus an overall aesthetic score, with short textual comments for each track. The authors use this dataset to benchmark a range of pretrained audio encoders and find that even the best models capture only partial aesthetic information: the top-performing audio encoder reaches a linear correlation of about 0.72 with human overall-score judgments, while a large multimodal language model given audio alone produces near-zero or negative correlation. The paper's central claim is that current audio representations are insufficient for human-aligned aesthetic assessment, but that human-written textual comments provide a strong supervisory signal that substantially closes the gap. When the language model is given comments and tags (with or without audio), correlation jumps to around 0.73, suggesting that aesthetic reasoning is currently carried more by language than by audio features.

Core claim

The paper establishes that the bottleneck for automatic music aesthetic assessment lies in audio representation quality, not in the availability of annotation dimensions or annotator agreement. Human comments encode compressed aesthetic reasoning that current audio models cannot extract from waveforms alone, as demonstrated by the sharp contrast between near-zero audio-only correlation and strong text-based correlation in zero-shot language-model probing. The dataset itself, with its multi-dimensional ratings and inter-annotator ICC values around 0.8, demonstrates that trained musicians can produce consistent multi-dimensional aesthetic judgments, making the gap a model limitation rather a标注

What carries the argument

The benchmark has three moving parts: (1) a 10-dimension annotation framework covering composition, arrangement, performance, and post-production stages, each rated on a 0–5 scale by about 10 trained annotators per track; (2) a CLAP-based audio-text alignment pipeline that fuses annotator comments and structured tags via a learnable gating mechanism before contrastive pre-adaptation, tested against frozen encoders (MuQ, MERT, CLAP variants) with a Transformer regression head; (3) a zero-shot probing setup using a multimodal LLM (Qwen2-Audio-7B) with three input configurations (audio-only, text-only, audio+text) to isolate modality contributions without dataset-specific training.

Load-bearing premise

The paper assumes that its 10 chosen perceptual dimensions comprehensively decompose music aesthetics into stable, generalizable criteria. If key aesthetic factors fall outside these dimensions, or if the dimensions overlap substantially rather than capturing independent aspects of perception, the benchmark's validity as a holistic evaluation tool would be compromised.

What would settle it

The strongest potential falsifier is the zero-shot audio-only LLM result: if a future model, trained on this dataset or a similar one, could predict multi-dimensional aesthetic scores from raw audio alone at correlation levels matching or exceeding text-based prediction, the paper's central claim about the audio representation bottleneck would be overturned.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the finding holds, future music aesthetic models will need to either develop audio encoders that internalize the perceptual reasoning currently captured only in text, or rely on hybrid pipelines where language models interpret audio features through aesthetic prompts.
  • The dataset's inclusion of AI-generated tracks (from Suno and Levo) alongside human-composed music means the benchmark can directly measure whether generative models are closing the aesthetic gap over time, providing a longitudinal evaluation tool for music generation research.
  • The strong performance of text-based supervision without aesthetic-score labels suggests that large-scale weakly-supervised pretraining on music reviews and comments could improve audio aesthetic models without requiring expensive multi-annotator rating data.
  • The finding that audio-only LLM probing yields near-zero correlation implies that current multimodal LLMs do not internally develop aesthetic representations from audio, challenging the assumption that scale alone will solve audio understanding.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If textual comments are the primary carrier of aesthetic signal, then the quality and granularity of comment collection may matter more than the number of annotators per track for future dataset construction; a dataset with fewer annotators but richer per-track reasoning text might outperform one with more ratings but terse comments.
  • The 10-dimension framework is organized around music production stages, but the paper does not test whether these dimensions are independent or whether some are redundant; a factor analysis of the rating matrix could reveal whether 10 dimensions collapse to fewer latent factors, which would simplify both annotation and modeling.
  • The near-zero audio-only LLM result is obtained without any task-specific fine-tuning, so it does not rule out the possibility that a multimodal LLM fine-tuned on aesthetic ratings could learn to extract aesthetic features from audio; the paper's zero-shot design isolates intrinsic modality informativeness but leaves the fine-tuned ceiling unmeasured.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The manuscript introduces MADB, a large-scale dataset for music aesthetic assessment comprising 9,999 tracks annotated by 30 trained musicians across 10 perceptual dimensions and one overall score, along with textual comments and tags. The authors establish a benchmark by evaluating several pretrained audio models (MuQ, MERT, CLAP variants) and a multimodal LLM (Qwen2-Audio) on the task of regressing aesthetic scores. The results indicate that current models capture only partial aesthetic information, with a substantial gap remaining to human-level judgment. The dataset is a valuable contribution, offering multi-dimensional, professional annotations that are currently lacking in the field.

Significance. The primary significance of this work lies in the dataset itself. The scale (9,999 tracks), the expertise of the annotators (30 trained musicians), and the multi-dimensional annotation framework aligned with the music production pipeline are substantial assets. The inclusion of textual comments and semantic tags further enriches the dataset, enabling multimodal analysis. The benchmarking results, particularly the near-zero correlation of audio-only LLM probing, provide a clear and falsifiable baseline for future research. The release of the dataset and code is a strong positive for reproducibility and community engagement.

major comments (2)
  1. Section 4.1 states that the dataset is randomly split into training and validation sets with a 'fixed ratio', but the ratio itself is not specified. This is a load-bearing methodological detail because the size of the validation set directly affects the variance of the reported metrics (MSE, LCC, SRCC, KRCC) in Table 2 and Table 3. Please specify the exact split ratio and, ideally, the number of samples in each set.
  2. Section 4.1 and Table 2: The evaluation uses 4 fixed random seeds for CLAP-based experiments, but only a single seed (42) for MuQ and MERT. Since MuQ is the best-performing model and is central to the claim that models capture 'only partial aesthetic information' (LCC=0.718 on the overall score), the absence of variance estimates for MuQ and MERT makes it difficult to assess the statistical significance of the performance gap between MuQ and the CLAP variants. Please run MuQ and MERT with multiple seeds and report the standard deviation, consistent with the CLAP experiments.
minor comments (4)
  1. Section 3.5 uses the abbreviation 'ICCK', while Appendix A.1 uses 'ICCk' and 'ICC'. Please standardize the abbreviation for Intraclass Correlation Coefficient.
  2. Section 3.2: 'Comments those less than 10 words will be ignored.' should be rephrased for grammatical correctness, e.g., 'Comments that are less than 10 words are ignored.'
  3. Table 2 is quite dense and difficult to read. Consider splitting it into two tables (one for MSE/LCC, one for SRCC/KRCC) or using a clearer layout to improve readability.
  4. Section 4.5 claims that the use of English comments 'proves the reliability of translations.' This is an overstatement; it merely suggests that the translated comments retain useful signal. Please soften this claim.

Circularity Check

0 steps flagged

No circularity found. The paper is a self-contained dataset and benchmark contribution.

full rationale

MADB is a dataset/benchmark paper with no derivation chain that could reduce to its inputs by construction. The annotation framework defines 10 perceptual dimensions and a separate overall score that is explicitly NOT a fixed aggregation of the dimensions (Section 3.1: 'This score is not defined as a fixed aggregation of individual dimensions, but instead reflects a higher-level perceptual integration process'). The evaluation framework trains standard regression heads on frozen pretrained encoders (MuQ, MERT, CLAP) using MSE loss on held-out data — no fitted parameter is relabeled as a prediction. The CLAP semantic adaptation explicitly avoids label leakage (Section 4.3: 'this adaptation is performed without using any aesthetic scores'). The LLM zero-shot probing experiment (Table 3) is a genuine modality comparison, not a tautological result. Self-citations to Jin et al. appear only in Related Work for context on prior aesthetic assessment approaches and do not form a load-bearing dependency for any claim in the current paper. The dataset's value rests on its scale (9,999 tracks), annotation quality (ICC≈0.8), and multi-dimensional structure, all of which are independently verifiable. No circularity patterns (self-definitional, fitted-input-as-prediction, self-citation load-bearing, uniqueness import, ansatz smuggling, or result renaming) are present.

Axiom & Free-Parameter Ledger

3 free parameters · 3 axioms · 1 invented entities

The paper's core contribution is empirical, relying on the construction of a dataset and the training of models. The free parameters are standard machine learning hyperparameters and architectural choices. The axioms are domain assumptions about music aesthetics and annotation quality. There are no invented theoretical entities.

free parameters (3)
  • Regression head architecture = Transformer-based
    The downstream regression head is a Transformer-based model, the specific architecture and hyperparameters of which are chosen for the task.
  • Learnable gating mechanism = Not specified
    The fusion of textual and tag representations uses a learnable gating mechanism, the parameters of which are fitted during training.
  • Training hyperparameters = Adam optimizer, fixed seeds
    Standard hyperparameters like learning rate and batch size are chosen for optimization, though specific values are not all detailed.
axioms (3)
  • domain assumption Music aesthetics can be decomposed into 10 orthogonal perceptual dimensions plus an overall score.
    This is the foundational assumption of the annotation framework in Section 3.1. The dimensions are designed to align with music production stages.
  • domain assumption Trained musicians provide more reliable and fine-grained aesthetic judgments than laypeople.
    The paper uses 30 professionally trained annotators, assuming their expertise yields higher quality data, as stated in Section 3.4.
  • ad hoc to paper Machine-translated comments preserve sufficient semantic information for model training.
    The paper translates Chinese comments to English using Qwen2.5-Instruct and uses them for CLAP training, assuming translation noise is acceptable, as mentioned in Section 3.2 and 5.
invented entities (1)
  • MADB dataset independent evidence
    purpose: To serve as a large-scale benchmark for music aesthetic assessment.
    The dataset is a constructed artifact, not a theoretical entity. Its existence and properties are verifiable through the released data on HuggingFace and GitHub.

pith-pipeline@v1.1.0-glm · 16028 in / 2039 out tokens · 232455 ms · 2026-07-09T22:43:04.208791+00:00 · methodology

0 comments
read the original abstract

Music aesthetic assessment is a challenging yet underexplored problem, requiring models to capture fine-grained, multi-dimensional human perceptual judgments. Progress in this area has been limited by the lack of large-scale datasets with structured aesthetic annotations. We introduce MADB, a large-scale dataset and benchmark comprising 9,999 tracks annotated by 30 trained annotators. Each track is rated by around 10 annotators across 10 perceptual dimensions and one overall score, with additional textual comments for multimodal analysis. We establish a unified evaluation framework over multiple pretrained models. Results reveal substantial gaps between model predictions and human judgments, exposing key limitations of current approaches. MADB provides a new benchmark for human-aligned music understanding. Project page: https://github.com/knownree/madb

Figures

Figures reproduced from arXiv: 2607.06929 by Duo Xu, Feng Yu, Haoxin Zhang, Jishang Chen, Liangke Zhao, Peiyang Yu, Sirui Zhang, Songchun Zhu, Tianle Wang, Xin Jin, Xinyi Tong.

Figure 1
Figure 1. Figure 1: Annotation examples Preprint. arXiv:2607.06929v1 [cs.SD] 8 Jul 2026 [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Histograms of average score across dimensions [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Three categories of evaluated methods: (1) pretrained encoders with post-training, (2) [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: presents the ICCK values across all perceptual dimensions. Overall, the agreement is consistently high, with most dimensions exceeding 0.80, indicating strong multi-rater reliability. Structurally grounded dimensions, such as melody_emotion and enunciation_and_singing_skill, achieve relatively higher agreement (e.g., above 0.83), suggesting that annotators share consistent judgments on well-defined percept… view at source ↗
Figure 5
Figure 5. Figure 5: Average overall score and standard deviation across data sources. [PITH_FULL_IMAGE:figures/full_fig_p014_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

18 extracted references · 18 canonical work pages · 16 internal anchors

  1. [1]

    MusicLM: Generating Music From Text

    URL https://arxiv.org/ abs/2301.11325. Shun Lei, Yaoxun Xu, Zhiwei Lin, Huaicheng Zhang, Wei Tan, Hangting Chen, Jianwei Yu, Yixuan Zhang, Chenyu Yang, Haina Zhu, Shuai Wang, Zhiyong Wu, and Dong Yu. Levo: High-quality song generation with multi-preference alignment,

    work page internal anchor Pith review Pith/arXiv arXiv

  2. [2]

    Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi, and Alexandre Défossez

    URL https://arxiv.org/abs/2503.08638. Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi, and Alexandre Défossez. Simple and controllable music generation,

    work page arXiv

  3. [3]

    Simple and Controllable Music Generation

    URL https://arxiv. org/abs/2306.05284. Xin Jin, Wu Zhou, Jingyu Wang, Duo Xu, and Yongsen Zheng. An order-complexity aesthetic assessment model for aesthetic-aware music recommendation, 2024a. URL https://arxiv. org/abs/2402.08300. Xin Jin, Wu Zhou, Jinyu Wang, Duo Xu, Yiqing Rong, and Jialin Sun. An order-complexity model for aesthetic quality assessment...

    work page internal anchor Pith review Pith/arXiv arXiv

  4. [4]

    An Order-Complexity Model for Aesthetic Quality Assessment of Homophony Music Performance

    URL https: //arxiv.org/abs/2304.11521. Xin Jin, Qianqian Qiao, Yi Lu, Shan Gao, Heng Huang, and Guangdong Li. Paintings and drawings aesthetics assessment with rich attributes for various artistic categories, 2024b. URL https: //arxiv.org/abs/2405.02982. Qianqian Qiao, DanDan Zheng, Yihang Bo, Bao Peng, Heng Huang, Longteng Jiang, Huaye Wang, Jingdong Che...

    work page internal anchor Pith review Pith/arXiv arXiv

  5. [5]

    VGA-Bench: A Unified Benchmark and Multi-Model Framework for Video Aesthetics and Generation Quality Evaluation

    URL https: //arxiv.org/abs/2604.10127. Cheng Liu, Hui Wang, Jinghua Zhao, Shiwan Zhao, Hui Bu, Xin Xu, Jiaming Zhou, Haoqin Sun, and Yong Qin. Musiceval: A generative music dataset with expert ratings for automatic text-to-music evaluation,

    work page internal anchor Pith review Pith/arXiv arXiv

  6. [6]

    MusicEval: A Generative Music Dataset with Expert Ratings for Automatic Text-to-Music Evaluation

    URLhttps://arxiv.org/abs/2501.10811. Jixun Yao, Guobin Ma, Huixin Xue, Huakang Chen, Chunbo Hao, Yuepeng Jiang, Haohe Liu, Ruibin Yuan, Jin Xu, Wei Xue, Hao Liu, and Lei Xie. Songeval: A benchmark dataset for song aesthetics evaluation,

    work page internal anchor Pith review Pith/arXiv arXiv

  7. [7]

    SongEval: A Benchmark Dataset for Song Aesthetics Evaluation

    URLhttps://arxiv.org/abs/2505.10793. Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding,

    work page internal anchor Pith review Pith/arXiv arXiv

  8. [8]

    Representation Learning with Contrastive Predictive Coding

    URLhttps://arxiv.org/abs/1807.03748. Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations,

    work page internal anchor Pith review Pith/arXiv arXiv

  9. [9]

    wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations

    URL https://arxiv.org/abs/ 2006.11477. 11 Haina Zhu, Yizhi Zhou, Hangting Chen, Jianwei Yu, Ziyang Ma, Rongzhi Gu, Yi Luo, Wei Tan, and Xie Chen. Muq: Self-supervised music representation learning with mel residual vector quantization,

    work page internal anchor Pith review Pith/arXiv arXiv 2006

  10. [10]

    URLhttps://arxiv.org/abs/2501.01108. Yizhi Li, Ruibin Yuan, Ge Zhang, Yinghao Ma, Xingran Chen, Hanzhi Yin, Chenghao Xiao, Chenghua Lin, Anton Ragni, Emmanouil Benetos, Norbert Gyenge, Roger Dannenberg, Ruibo Liu, Wenhu Chen, Gus Xia, Yemin Shi, Wenhao Huang, Zili Wang, Yike Guo, and Jie Fu. Mert: Acoustic music understanding model with large-scale self-s...

    work page internal anchor Pith review Pith/arXiv arXiv

  11. [11]

    MERT: Acoustic Music Understanding Model with Large-Scale Self-supervised Training

    URL https://arxiv.org/abs/2306.00107. Qiuqiang Kong, Yin Cao, Turab Iqbal, Yuxuan Wang, Wenwu Wang, and Mark D. Plumbley. Panns: Large-scale pretrained audio neural networks for audio pattern recognition,

    work page internal anchor Pith review Pith/arXiv arXiv

  12. [12]

    PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition

    URL https://arxiv.org/abs/1912.10211. Ke Chen, Xingjian Du, Bilei Zhu, Zejun Ma, Taylor Berg-Kirkpatrick, and Shlomo Dubnov. Hts-at: A hierarchical token-semantic audio transformer for sound classification and detection,

    work page internal anchor Pith review Pith/arXiv arXiv 1912

  13. [13]

    HTS-AT: A Hierarchical Token-Semantic Audio Transformer for Sound Classification and Detection

    URL https://arxiv.org/abs/2202.00874. Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Marianna Nezhurina, Taylor Berg-Kirkpatrick, and Shlomo Dubnov. Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation,

    work page internal anchor Pith review Pith/arXiv arXiv

  14. [14]

    URLhttps://arxiv.org/abs/2211.06687. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision,

    work page internal anchor Pith review Pith/arXiv arXiv

  15. [15]

    Learning Transferable Visual Models From Natural Language Supervision

    URL https: //arxiv.org/abs/2103.00020. Shangda Wu, Zhancheng Guo, Ruibin Yuan, Junyan Jiang, Seungheon Doh, Gus Xia, Juhan Nam, Xiaobing Li, Feng Yu, and Maosong Sun. Clamp 3: Universal music information retrieval across unaligned modalities and unseen languages. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and {Mohammad Taher} Pilehvar, editors,Fin...

    work page internal anchor Pith review Pith/arXiv arXiv

  16. [16]

    doi: 10.18653/v1/ 2025.findings-acl.133. Publisher Copyright: © 2025 Association for Computational Linguistics.; 63rd Annual Meeting of the Association for Computational Linguistics, ACL 2025 ; Conference date: 27-07-2025 Through 01-08-2025. Zihao Wang, Shuyu Li, Tao Zhang, Qi Wang, Pengfei Yu, Jinyang Luo, Yan Liu, Ming Xi, and Kejun Zhang. Muchin: A chi...

    work page doi:10.18653/v1/ 2025

  17. [17]

    URLhttps://arxiv.org/abs/2402.09871. Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin...

    work page internal anchor Pith review Pith/arXiv arXiv

  18. [18]

    Qwen2.5 Technical Report

    URL https://arxiv.org/abs/2412.15115. 12 A Technical appendices and supplementary material A.1 Inter-Annotator Agreement Across Dimensions Figure 4 presents the ICCK values across all perceptual dimensions. Overall, the agreement is consistently high, with most dimensions exceeding 0.80, indicating strong multi-rater reliability. Structurally grounded dim...

    work page internal anchor Pith review Pith/arXiv arXiv