REVIEW 4 major objections 6 minor 29 references
A widely used music-aesthetics reward model, SongEval, mistakes genre for quality: its predicted scores track pop-likeness (Spearman up to 0.88), and a focal plus group-regularized training objective reduces that shortcut and improves human
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-02 03:20 UTC pith:F6AYS7JY
load-bearing objection A solid diagnostic showing SongEval's pop-axis bias, attached to a mitigation that needs more statistical rigor and a harder look at whether genre invariance is actually the right target. the 4 major comments →
Genre Bias or Aesthetic Perception? Identifying and Mitigating Shortcut Learning in Music Evaluation
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The core claim is that the empirical risk minimizer for aesthetic scoring uses genre as a shortcut whenever the training distribution makes genre predictive of the label after fixing quality-relevant features. The paper formalizes this with a conditional-independence target: under the population distribution, the aesthetic score should be independent of genre given quality; in the training data, that independence is violated. Diagnostic evidence includes significant genre effects in genre-balanced probes, asymmetric pairwise preference errors concentrated on pop-versus-jazz/classical pairs, and a Spearman rank correlation of 0.81–0.88 between predicted scores and a learned pop-proximity dire
What carries the argument
The load-bearing mechanism is a training objective L = focal + λ_group · L_group. Focal loss uses e_i^{γ+1}, the regression error raised to a power, to re-weight sample gradients toward hard examples instead of letting easy, shortcut-aligned samples dominate. The group regularizer computes, per genre, an exponential-moving-average performance deviation Δ_k = μ_k − μ, and penalizes (g_k − μ)^2 with weight 1+|Δ_k|, encouraging balanced learning across genres. This is supported by a diagnostic probe: a lightweight pop/non-pop binary classifier whose logits serve as a pop-likeness score, used to measure how tightly predictions align with the shortcut direction.
Load-bearing premise
The chain rests on the assumption that aesthetic judgment should be genre-blind once musical quality is fixed—that a listener's score should not change when they learn a track's genre—which the paper does not test directly.
What would settle it
Gather a genre-balanced corpus with matched production quality, have expert raters score each track twice (once with genre labels visible, once hidden), and check whether ratings shift with the label; if they do, the independence assumption fails and the debiasing objective could be correcting a difference that humans genuinely perceive.
If this is right
- Reward models trained this way should rank cross-genre pairs more like humans, reducing systematic overestimation of pop and undervaluation of jazz and classical.
- Music-generation systems optimized with the corrected scores should be less steered toward pop-like production, helping preserve genre diversity in output.
- Because same-genre preference accuracy also improves (0.669→0.703), the fix appears to sharpen genuine quality discrimination, not just rebalance genre labels.
- The measurable drop in pop-likeness correlation (from 0.81–0.88 to 0.62–0.68) provides a simple audit signal for whether a music reward model has this shortcut.
- The method adds no new data or architecture—only a modified loss with two tunable coefficients—so it can be applied to existing aesthetic scorers.
Where Pith is reading between the lines
- If genre is a legitimate part of aesthetic preference—listeners who know a genre may genuinely rate it higher—then the paper's independence assumption is too strong, and the correction could move predictions away from human judgment in some populations. A rating study that holds quality fixed while varying listener-genre familiarity would test this.
- The same diagnostic pattern (train a probe for a suspected shortcut direction, measure rank correlation with scores) could be reused to audit other audio or text reward models for shortcuts such as loudness, instrumentation, or era before deployment.
- The focal+group objective is a generic debiasing recipe; it could be applied to other spurious subpopulations (e.g., recording quality, language, artist identity) whenever group labels are available, though the paper only evaluates genre.
- If the technique is adopted widely, evaluation suites for music generation could report genre-disaggregated preference accuracy alongside overall accuracy, making shortcut bias visible rather than hidden in aggregate scores.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper argues that SongEval, a music aesthetics reward model, learns a genre shortcut: its predicted scores are strongly aligned with a pop-related direction (Spearman 0.81–0.88) and it systematically overestimates pop while undervaluing jazz and classical. The authors diagnose this via training-data statistics, balanced genre sets, CMI pairwise preference comparisons, and pop-proximity probes. They then propose a training objective combining a focal regression loss with a group-level regularization term that penalizes per-genre performance deviations from the global mean, and report that it reduces the pop-proximity correlation, makes training trajectories more synchronized across genres, and improves CMI-Pref accuracy from 0.687 to 0.715. The paper concludes that genre-induced shortcut learning can be mitigated while improving alignment with human preferences.
Significance. If the central claim is correct, the paper addresses a practically important failure mode for reward models used in music generation pipelines, and the diagnostic pipeline—training-data inspection, balanced-set probes, external human-preference benchmarking, and pop-proximity analysis—is a useful template for studying shortcut learning in audio evaluation. The proposal is simple and the use of an external human-preference benchmark is a strength. However, the paper's normative target, Eq. (1), is an untested assumption that is in tension with the paper's own human-preference data, and the reported mitigation gains are small, unreplicated, and partly inconsistent across pairs. The paper would need to substantially clarify or revise its target assumption and provide statistical reliability evidence before its main claims can be accepted.
major comments (4)
- [§2, Eq. (1); §4, Eq. (16); §5.3, Tables 2–3] The entire 'shortcut' framing and the proposed mitigation rest on the conditional-independence assumption y ⊥ (z_g, z_r) | z_q, i.e., that genre should not influence aesthetic score once quality is fixed. This assumption is never tested, and the paper's own human-preference data contradict a strong genre-neutral view: e.g., Table 2/3 shows GT win rate for rock vs. hip-hop of 0.893 and for jazz vs. rock of 0.750, indicating substantial genre-linked human preferences. Under the proposed objective, group regularization (Eq. 16) directly pushes predictions toward genre-invariant error, which can move predictions away from human judgments. Concretely, on pop–rock the predicted win rate moves from 0.774 (baseline) to 0.806 (ours) while GT is 0.645; on pop–classical it moves from 0.807 to 0.843 while GT is 0.675. The claim that these changes represent bias reduction is therefore not established
- [§5.3, Tables 2–3 and text] The reported gains are small (overall CMI-Pref 0.687→0.715; CMI-Pref-Pseudo 0.549→0.566) and no confidence intervals, number of seeds, or significance tests are reported. Batch size is 1 (Sec. 5.1), so the EMA group statistics are noisy; without seed-level variance or bootstrap intervals it is unclear whether the differences are reproducible. Moreover, pair-level accuracies regress on several pairs, e.g., pop–jazz 0.879→0.848 and jazz–classical 0.700→0.691. The text says 'well-aligned pairs such as pop–hiphop remain stable' (0.795→0.826), but the overall pattern is a redistribution of errors, not a consistent improvement. The ablation (Sec. 5.3) reports only overall numbers, so the individual contributions of focal loss and group regularization to specific pairs cannot be assessed.
- [§3.3, Eq. (8)] The 'controlled' evaluation in Sec. 3.3 balances only the number of samples per genre, not the quality distributions. Equal sample counts do not ensure equal aesthetic quality, so the observed genre–score gaps could partly reflect genuine quality differences. The paper states that the design 'reduces two major confounders: genre imbalance and variation in production quality,' but no operationalization of production quality is given; without a quality-matched control (e.g., matching on human-rated quality or on an independent quality measure), the ANOVA and mean-score gaps do not by themselves prove that the model is using genre as a shortcut.
- [§4, Eq. (16) and Fig. 3] The 'more synchronized' training trajectories in Fig. 3 are partly an artifact of the objective: Eq. (16) directly penalizes (g_k − μ)² and Δ_k = μ_k − μ, so reducing across-genre EMA spread is, by construction, what the loss is optimized to do. Presenting this as independent evidence that the model 'learns more uniformly across genres' is circular. The external CMI-Pref benchmark is the appropriate non-circular test, but as noted above its results are mixed and lack statistical support.
minor comments (6)
- [§1, Contributions] The phrase 'futher direction analysis' should be 'further direction analysis'.
- [§3.3] 'ANOV A' contains an errant space; should be 'ANOVA'.
- [Table 3 / §5.3] The caption and text say 'improves consistently' on challenging pairs, but Table 3 shows decreased accuracy on pop–jazz and jazz–classical. Please reconcile the wording with the numbers.
- [§5.1] Hyperparameters γ=2 and λ_group=1e6 are fixed without sensitivity analysis. Since λ_group is extremely large, the group constraint likely dominates; a small sensitivity study would help interpret the method.
- [§3.1] The use of Qwen3-Omni for genre labeling is validated only by overall agreement (>90%). It would be useful to report per-genre agreement, especially for jazz/classical, which drive the main findings.
- [§2] The notation z=(z_q,z_g,z_r) and the Bayes predictor in Eq. (3) are clear, but the step 'E_p[y|z] = E_p[y|z_q]' relies on the strong CI assumption and is not merely a notational convenience; please flag it as an assumption rather than a definition.
Circularity Check
Bias-gap reduction is partly built into the proposed loss; the CMI preference improvements are external and keep the central claim from being fully circular.
specific steps
-
self definitional
[Sec. 5.3, Eqs. (14) and (16), Fig. 3]
"∆k = µk − µ. ... Lgroup = 1/|G| Σ_k w_k·(g_k − μ)^2. ... under the proposed objective, the EMA trajectories become significantly more synchronized across genres, different genres progress at a more consistent rate, and the gap between fast- and slow-learning genres is substantially reduced."
The bias is operationalized as the group-performance gap ∆k = µk − µ in Eq. (14), and the proposed mitigation in Eq. (16) is a squared penalty on exactly that gap. Consequently, the 'synchronized' group trajectories in Fig. 3 are the regularizer achieving its own target, not an independent empirical confirmation that genre bias is reduced. The paper presents this gap shrinkage as evidence that the method 'reduces genre-dependent bias,' but that particular result is definitionally tied to the objective. However, the CMI-Pref comparisons use an external human-preference benchmark, so the circularity is partial rather than total.
full rationale
The only construction-reducing step I can exhibit is the use of the group-performance gap as both the diagnostic of bias and the quantity directly penalized by Lgroup; Fig. 3 is therefore partly an illustration of the loss working as specified. The Section 2 assumption y ⊥ (z_g, z_r) | z_q is a normative modeling premise, not a circular derivation, and its validity is a separate empirical question. I found no load-bearing self-citations, imported uniqueness theorems, or ansatz-by-citation. The central claim about improved alignment with human preferences is supported by the external CMI benchmark (0.687→0.715 on CMI-Pref) and by an independently trained pop-likeness classifier, so the paper is not self-contained in a circular way. Score 4 reflects the localized partial circularity without overstating it.
Axiom & Free-Parameter Ledger
free parameters (3)
- gamma (focal focusing strength) =
2
- lambda_group (group regularization coefficient) =
1e6
- EMA decay beta =
not reported
axioms (5)
- standard math Squared-loss Bayes predictor and conditional mutual information formalism (Eqs. 3, 7).
- domain assumption Aesthetic judgments are genre-independent after conditioning on quality-relevant factors (y ⊥ (z_g,z_r) | z_q under the population).
- domain assumption Qwen3-Omni genre labels are a reliable proxy with >90% agreement on MTG-Jamendo and M6; same reliability is assumed on CMI where agreement is not reported.
- domain assumption The balanced MTG-Jamendo and M6 subsets reduce 'variation in production quality' confound.
- domain assumption SongEval's five dimensions are redundant (all pairwise correlations >0.95) and averaging yields a valid unified aesthetic measure.
read the original abstract
Music aesthetics scoring plays a critical role in applications such as dataset curation, generative model evaluation, and reward modeling for music generation. Recent approaches rely on deep neural networks trained on human-annotated ratings, but these models may exploit spurious correlations rather than capturing perceptually meaningful aesthetics. In this work, we identify a previously underexplored failure mode in music evaluation models: genre-induced shortcut learning. Through a systematic analysis of SongEval, we show that biases in training data lead to strong correlations between genre-related features and predicted scores, causing the model to use them as a proxy for aesthetics. This results in systematic overestimation of pop music and undervaluation of high-quality samples from other genres, leading to predictions that are inconsistent with human preferences. To address this issue, we propose a training objective that jointly reweights hard samples and regularizes group-level performance, encouraging the model to learn genre-invariant representations of musicality. Experimental results demonstrate that our method reduces genre-dependent bias and improves alignment with human preferences, as reflected by gains in both cross-genre and within-genre preference alignment.
Reference graph
Works this paper leans on
-
[1]
INTRODUCTION Automatic music aesthetics assessment plays an increas- ingly important role in modern music generation and eval- uation pipelines [1]. Music aesthetics scores are widely used for filtering and curating datasets, benchmarking mu- sic generation models [2], and providing reward signals to align generative systems with human preferences [3]. Re...
Pith/arXiv arXiv 2026
-
[2]
Ideally, aesthetic judgments should depend on genre only through quality-relevant musical factors
PROBLEM SETUP Letz= (z q, zg, zr)denote the representation of a mu- sic sample, wherez q captures quality-relevant factors,z g encodes genre identity, andz r denotes other residual vari- ation. Ideally, aesthetic judgments should depend on genre only through quality-relevant musical factors. We there- fore assume that, under the population distributionp, ...
-
[3]
DIAGNOSTIC ANALYSIS We conduct a series of diagnostic analyses on SongEval [5], a widely used music aesthetics evalua- tion model [14–18]. Accordingly, we analyze genre- related shortcut learning through a progressive diagnostic pipeline: first verifying spurious genre–score correlations in the training data; then testing whether these shortcuts transfer ...
2026
-
[4]
Instead of relying purely on intrin- sic acoustic cues relevant to musical aesthetics, the model tends to exploit correlations between genre-related features and score
PROPOSED METHOD Section 3 shows that SongEval exhibits obvious shortcut learning behavior. Instead of relying purely on intrin- sic acoustic cues relevant to musical aesthetics, the model tends to exploit correlations between genre-related features and score. As a result, samples associated with domi- nant patterns become easier to predict, while others r...
2026
-
[5]
Following the standard SongEval setup, the baseline model is trained us- ing the MSE objective with the Adam optimizer, a learn- ing rate of3×10 −5, for 30 epochs
EXPERIMENTS 5.1 Experimental Setup We evaluate the proposed method under the same data and evaluation protocol described in Section 3. Following the standard SongEval setup, the baseline model is trained us- ing the MSE objective with the Adam optimizer, a learn- ing rate of3×10 −5, for 30 epochs. Due to the use of full-length audio inputs, the batch size...
2026
-
[6]
Through a pro- gressive diagnostic analyses, we show that models rely on genre-related signals as proxies for musical aesthetics, fur- ther leading to pop-centric bias
CONCLUSION We identify a critical failure mode in music aesthetics eval- uation: genre-induced shortcut learning. Through a pro- gressive diagnostic analyses, we show that models rely on genre-related signals as proxies for musical aesthetics, fur- ther leading to pop-centric bias. We further demonstrate that this behavior is propagated in shortcut-driven...
-
[7]
While our method mitigates shortcut learning dur- ing optimization, a more diverse dataset with reliable and balanced annotations remains essential
LIMITA TIONS First, model performance is fundamentally constrained by data. While our method mitigates shortcut learning dur- ing optimization, a more diverse dataset with reliable and balanced annotations remains essential. Second, beyond genre, models may still exploit other spurious cues (e.g., loudness or instrumentation). Iden- tifying and characteri...
-
[8]
Survey on the evaluation of generative models in music,
A. Lerch, C. Arthur, N. Bryan-Kinns, C. Ford, Q. Sun, and A. Vinay, “Survey on the evaluation of generative models in music,”ACM Computing Surveys, vol. 58, no. 4, pp. 1–36, 2025
2025
-
[9]
Benchmarking music generation models and metrics via human preference studies,
F. Grötschla, A. Solak, L. A. Lanzendörfer, and R. Wat- tenhofer, “Benchmarking music generation models and metrics via human preference studies,” inICASSP 2025-2025 IEEE International Conference on Acous- tics, Speech and Signal Processing (ICASSP). IEEE, 2025, pp. 1–5
2025
-
[10]
Musicrl: aligning Proceedings of the 27th ISMIR Conference, Abu Dhabi, UAE, November 08–12, 2026 music generation to human preferences,
G. Cideron, S. Girgin, M. Verzetti, D. Vincent, M. Kastelic, Z. Borsos, B. McWilliams, V . Ungure- anu, O. Bachem, O. Pietquinet al., “Musicrl: aligning Proceedings of the 27th ISMIR Conference, Abu Dhabi, UAE, November 08–12, 2026 music generation to human preferences,” inProceed- ings of the 41st International Conference on Machine Learning, 2024, pp. 8968–8984
2026
-
[11]
Meta audiobox aesthetics: Unified automatic qual- ity assessment for speech, music, and sound,
A. Tjandra, Y .-C. Wu, B. Guo, J. Hoffman, B. Ellis, A. Vyas, B. Shi, S. Chen, M. Le, N. Zacharovet al., “Meta audiobox aesthetics: Unified automatic qual- ity assessment for speech, music, and sound,”arXiv preprint arXiv:2502.05139, 2025
Pith/arXiv arXiv 2025
-
[12]
Songeval: A benchmark dataset for song aesthetics evaluation,
J. Yao, G. Ma, H. Xue, H. Chen, C. Hao, Y . Jiang, H. Liu, R. Yuan, J. Xu, W. Xueet al., “Songeval: A benchmark dataset for song aesthetics evaluation,” arXiv preprint arXiv:2505.10793, 2025
Pith/arXiv arXiv 2025
-
[13]
Musiceval: A generative music dataset with expert ratings for automatic text-to- music evaluation,
C. Liu, H. Wang, J. Zhao, S. Zhao, H. Bu, X. Xu, J. Zhou, H. Sun, and Y . Qin, “Musiceval: A generative music dataset with expert ratings for automatic text-to- music evaluation,” inICASSP 2025-2025 IEEE Inter- national Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2025, pp. 1–5
2025
-
[14]
Fr\’echet audio distance: A reference-free metric for evaluating music enhancement algorithms,
D. Roblek, K. Kilgour, M. Sharifi, and M. Zuluaga, “Fr\’echet audio distance: A reference-free metric for evaluating music enhancement algorithms,” inProc. Interspeech, 2019, pp. 2350–2354
2019
-
[15]
Mulan: A joint embedding of music audio and natural language,
Q. Huang, A. Jansen, J. Lee, R. Ganti, J. Y . Li, and D. P. Ellis, “Mulan: A joint embedding of music audio and natural language,” inIsmir 2022 Hybrid Conference, 2022
2022
-
[16]
Adapting frechet audio distance for generative music evaluation,
A. Gui, H. Gamper, S. Braun, and D. Emmanouilidou, “Adapting frechet audio distance for generative music evaluation,” inICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Process- ing (ICASSP). IEEE, 2024, pp. 1331–1335
2024
-
[17]
Yue: Scaling open foundation models for long-form music genera- tion,
R. Yuan, H. Lin, S. Guo, G. Zhang, J. Pan, Y . Zang, H. Liu, Y . Liang, W. Ma, X. Duet al., “Yue: Scaling open foundation models for long-form music genera- tion,”arXiv preprint arXiv:2503.08638, 2025
arXiv 2025
-
[18]
From aesthetics to human preferences: Compara- tive perspectives of evaluating text-to-music systems,
H. Zhang, J. Liang, H. Phan, W. Wang, and E. Bene- tos, “From aesthetics to human preferences: Compara- tive perspectives of evaluating text-to-music systems,” in2025 IEEE 35th International Workshop on Machine Learning for Signal Processing (MLSP). IEEE, 2025, pp. 1–6
2025
-
[19]
The icassp 2026 automatic song aesthetics evaluation challenge,
G. Ma, Y . Xia, J. Yao, H. Xue, H. Liu, S. Wang, H. Liu, and L. Xie, “The icassp 2026 automatic song aesthetics evaluation challenge,”arXiv preprint arXiv:2601.07237, 2026
arXiv 2026
-
[20]
Robust learning from noisily labeled long- tailed data via fairness regularizer,
J. Wei, Z. Zhu, G. Niu, T. Liu, S. Liu, M. Sugiyama, and Y . Liu, “Robust learning from noisily labeled long- tailed data via fairness regularizer,” inProceedings of the AAAI Conference on Artificial Intelligence, vol. 40, no. 42, 2026, pp. 35 847–35 856
2026
-
[21]
Diffrhythm+: Controllable and flexible full-length song genera- tion with preference optimization,
H. Chen, Y . Jiang, G. Ma, C. Hao, S. Wang, J. Yao, Z. Ning, M. Meng, J. Luan, and L. Xie, “Diffrhythm+: Controllable and flexible full-length song genera- tion with preference optimization,”arXiv preprint arXiv:2507.12890, 2025
Pith/arXiv arXiv 2025
-
[22]
Ace- step: A step towards music generation foundation model,
J. Gong, S. Zhao, S. Wang, S. Xu, and J. Guo, “Ace- step: A step towards music generation foundation model,”arXiv preprint arXiv:2506.00045, 2025
Pith/arXiv arXiv 2025
-
[23]
Levo: High- quality song generation with multi-preference align- ment,
S. Lei, Y . Xu, H. Zhang, H. Chen, Y . Zhang, C. Yang, H. Zhu, S. Wang, Z. Wu, D. Yuet al., “Levo: High- quality song generation with multi-preference align- ment,” inThe Thirty-ninth Annual Conference on Neu- ral Information Processing Systems
-
[24]
Jam: A tiny flow-based song generator with fine-grained controllability and aesthetic alignment,
R. Liu, C.-Y . Hung, N. Majumder, T. Gautreaux, A. A. Bagherzadeh, C. Li, D. Herremans, and S. Poria, “Jam: A tiny flow-based song generator with fine-grained controllability and aesthetic alignment,”arXiv preprint arXiv:2507.20880, 2025
Pith/arXiv arXiv 2025
-
[25]
Songecho: Towards cover song generation via instance-adaptive element-wise linear modulation,
S. Li, Y . Li, Z. Wang, Y . Zhang, F. Wu, O. Deussen, T.- Y . Lee, and W. Dong, “Songecho: Towards cover song generation via instance-adaptive element-wise linear modulation,”arXiv preprint arXiv:2602.19976, 2026
arXiv 2026
-
[26]
The mtg-jamendo dataset for automatic mu- sic tagging
D. Bogdanov, M. Won, P. Tovstogan, A. Porter, and X. Serra, “The mtg-jamendo dataset for automatic mu- sic tagging.” ICML, 2019
2019
-
[27]
M6: multi- generator, multi-domain, multi-lingual and cultural, multi-genres, multi-instrument machine-generated mu- sic detection databases,
Y . Li, H. Li, L. Specia, and B. Schuller, “M6: multi- generator, multi-domain, multi-lingual and cultural, multi-genres, multi-instrument machine-generated mu- sic detection databases,”Scientific Reports, 2026
2026
-
[28]
Cmi- rewardbench: Evaluating music reward models with compositional multimodal instruction,
Y . Ma, H. Xia, H. Gao, W. Chen, Y . Ye, Y . Yang, S. Chang, M. Ding, Y . Li, R. Yuanet al., “Cmi- rewardbench: Evaluating music reward models with compositional multimodal instruction,”arXiv preprint arXiv:2603.00610, 2026
Pith/arXiv arXiv 2026
-
[29]
J. Xu, Z. Guo, H. Hu, Y . Chu, X. Wang, J. He, Y . Wang, X. Shi, T. He, X. Zhuet al., “Qwen3-omni technical report,”arXiv preprint arXiv:2509.17765, 2025
Pith/arXiv arXiv 2025
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.