Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Do Captioning Metrics Reflect Music Semantic Alignment?

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper shows that BLEU, METEOR, ROUGE, SPICE, and FENSE barely track listener judgments of music captions, scoring paraphrases lower than distorted rewrites.

desk verdict A plausible warning about n-gram metrics for music captioning, but the correlation evidence is too under-powered and inconsistent to carry the central claim. read the letter →

arxiv 2411.11692 v1 pith:NX7G4B2X submitted 2024-11-18 cs.SD cs.IReess.AS

Jinwoo Lee , Kyogu Lee This is my paper · ORCID
classification cs.SDcs.IReess.AS
keywords musiccaptioningevaluationmetricsn-gramoverlapsemanticalignmenthumanCapsFENSEmeanopinionscore
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the text-overlap metrics commonly used to evaluate music captions do not measure whether a caption is semantically aligned with the audio. In a human listening study on the MusicCaps evaluation set, the authors find that paraphrased captions with similar human quality ratings receive much lower BLEU, METEOR, ROUGE, and SPICE scores than the originals, while semantically distorted captions can score higher. They also report that all five tested metrics, including the audio-specific FENSE, have near-zero Pearson correlation with human mean opinion scores (around 0.08). The point is that current automated scores can mislead researchers about which captioning models actually produce descriptions people agree with.

What carries the argument

The load-bearing mechanism is the contrast between three caption types, original, paraphrased, and distorted, measured against human mean opinion scores. Paraphrases hold semantics roughly constant while changing surface wording; distorted captions hold syntax roughly constant while changing musical facts. Running standard n-gram and embedding-based metrics over these pairs exposes which scores track wording rather than meaning, and correlating metric scores with human MOS across MusicCaps quantifies how far the metrics depart from listener judgment.

What would settle it

Run a larger human-rating study, for example 100 or more clips with all caption types rated by fresh listeners, and check whether n-gram metrics correlate positively with human MOS; if BLEU, METEOR, or ROUGE show robust correlation, or if humans rate the distorted captions as good as the paraphrases, the paper's conclusion would not hold.

Watch

Extended reading notes

Core claim

The central claim is that existing captioning metrics are overly sensitive to syntactic variation and lack alignment with actual human evaluations when applied to music captions. Using 50 listeners and 30 sampled clips from MusicCaps, the authors compare human mean opinion scores with metric scores for original, model-generated, and paraphrased captions. N-gram metrics such as BLEU, METEOR, and ROUGE fall sharply for paraphrases that human raters judge about as good as the originals, and distorted captions that preserve syntax but change musical content receive higher overlap scores than meaning-preserving paraphrases. The correlation table (20 entries) shows all metrics, including FENSE, sit near zero, leading the authors to conclude that the field needs a reevaluation of how music captions are scored.

Load-bearing premise

The entire argument depends on the authorship judgment that the paraphrased captions truly mean the same as the originals and the distorted captions do not, a judgment that is not independently verified by human ratings for the distorted captions.

Editorial extensions

If this is right

  • If these results hold, published comparisons of music-captioning models that rely on BLEU or ROUGE may rank models by wording similarity rather than by perceived caption quality.
  • A metric that rewards synonyms and restructured sentences while checking musical content, such as genre, instruments, and mood, would be needed to replace n-gram overlap.
  • Distorted captions scoring above paraphrases means n-gram metrics can be gamed by outputting fluent-looking text that repeats reference syntax, so automated leaderboards should not be treated as quality measures.
  • Even FENSE, which was designed with audio captions in mind, does not align with human judgments on music captions, so the problem is not solved simply by borrowing a newer NLP metric.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural next step, not run in the paper, is to test whether a semantic metric built from music-tag or metadata categories (genre, instrument, mood) correlates with human MOS more strongly than FENSE does.
  • If the near-zero correlations generalize beyond MusicCaps, then other audio-captioning benchmarks may need similar human-validation studies before their automatic scores are trusted.
  • Because the paper's correlation analysis uses 20 entries rather than the 30 sampled clips, a larger replication study would give a sharper estimate of the true correlation values and test whether the near-zero result is stable.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. Do Captioning Metrics Reflect Music Semantic Alignment? (Lee and Lee, ISMIR 2024 late-breaking abstract) investigates whether standard captioning metrics (BLEU, METEOR, ROUGE, SPICE, FENSE) reflect human semantic judgments of music captions. The authors sampled 30 clips from the MusicCaps evaluation set, generated three caption types per clip (original ground truth, LP-MusicCaps inference, and author-authored paraphrases), and collected MOS ratings from 50 MTurk participants. They report metric scores for original, paraphrased, and distorted captions (Table 1), a bar plot of average scores (Figure 1), and Pearson correlations between each metric and MOS for 20 entries (Table 2). The paper concludes that n-gram metrics are overly sensitive to syntactic variation and lack alignment with human evaluations.

Significance. The question is timely: music captioning is growing, and the field needs to know whether borrowed NLP metrics are valid. The paper provides concrete examples (Table 1) that nicely illustrate how paraphrases are penalized by n-gram overlap, and it attempts a human-subject correlation study. If the correlation results were statistically well-grounded, the paper would make a valuable contribution to the evaluation methodology discussion. As it stands, the evidentiary basis for the central quantitative claim is too weak, so the paper functions more as a motivating demonstration than as a demonstrated finding.

major comments (4)
  1. [Section 3.2, Table 2] The correlation analysis reports Pearson r values for '20 entries' although Section 2 states that 30 audio clips were sampled. The manuscript does not explain the discrepancy (e.g., exclusions, missing ratings, or data filtering). With n=20, the 95% confidence interval for r=0.08 spans roughly [-0.38, 0.51], so the data are statistically indistinguishable from zero and also from moderately positive or negative correlations. The statement that metrics 'do not show significant correlations' is not supported without p-values, confidence intervals, or an equivalence test that bounds the plausible correlation. This is the central numerical evidence for the 'lack alignment' claim, so it is load-bearing.
  2. [Section 3.1, Table 1] The distorted captions are labeled 'Semantic X, Syntactic O' in Table 1, but no human MOS values are provided for them, and no validation is reported that the distortions are semantically incorrect while the paraphrases are semantically equivalent. The claim that 'most n-gram-based metrics tend to favor distorted captions over paraphrased captions' depends on these unverified labels; without a human ground-truth check on the distorted captions, the comparison cannot be interpreted as evidence about semantic alignment. This also weakens the syntactic/semantic distinction that motivates the paper.
  3. [Section 3.1, Figure 1] The text says that evaluation metrics 'show a significant decrease' when comparing original to paraphrased captions, but no statistical test, confidence interval, or effect size is reported. Figure 1 appears to show only mean values, with no error bars or per-clip data. A paired significance test (e.g., Wilcoxon signed-rank test on per-caption metric differences) is needed before concluding that the metrics are systematically sensitive to paraphrasing.
  4. [Section 3.2, Table 2] The correlation is computed over pooled observations that are not independent: each audio clip contributes multiple caption types (original, inference, paraphrased), and the paraphrased captions were deliberately constructed to have low n-gram overlap while receiving high human ratings. This clustering and range restriction can attenuate Pearson r. The authors should report per-clip correlations, a mixed-effects model, or at least a correlation that accounts for the repeated-measures structure. Without this, the numerical r-values do not establish that the metrics 'lack alignment' with human judgments.
minor comments (5)
  1. [Section 3.2, Table 2] Report p-values or confidence intervals, and specify the exact number of observations and why it differs from the 30 clips in Section 2.
  2. [Figure 1] Add error bars and indicate the number of captions per bar; also annotate which differences are statistically significant.
  3. [Section 2] Provide details on participant recruitment and filtering, how many ratings each caption received, and the aggregation method for MOS (e.g., mean vs. median).
  4. [Conclusion] The conclusion 'we demonstrate existing metrics are overly sensitive to syntactic variations' should be softened to 'provide evidence consistent with' given the small sample and the lack of statistical testing.
  5. [References] Reference [10] is the MusicLM paper; consider citing the MusicCaps dataset (or its specific description) directly for the dataset details.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper is an empirical evaluation whose claims rest on external human MOS ratings and public metrics, not on fitted parameters or self-citation.

full rationale

The paper makes no derivation that reduces a predicted quantity to an input. It reports correlations between standard captioning metrics (BLEU, METEOR, ROUGE, SPICE, FENSE) and human MOS collected from MTurk; the metrics are externally defined and the human ratings are independent ground truth. The paraphrased-caption examples are hand-authored stimuli, not quantities fitted from the metrics, so the observed drop in n-gram scores is an empirical finding rather than a construction artifact. The citation to LP-MusicCaps (Ref. [2]) supplies the captioning model used to generate one caption type, but the paper's central claim about metric limitations does not depend on that model's correctness. Statistical weaknesses noted by a skeptical reader (n=20, no confidence intervals, no significance tests, possible non-independence across captions per clip) undermine the strength of the correlation conclusion, but they are concerns about statistical inference and evidence quality, not circularity. No step in the paper equates a conclusion to its assumptions by definition or by a fitted parameter, so the circularity score is 0.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No fitted parameters or new entities. The main epistemic burden is the untested assumption that paraphrased and distorted captions have the semantic statuses the authors assign, plus the unexplained 20 vs 30 discrepancy.

assumptions (3)
  • domain assumption Human MOS scores from 50 MTurk participants are a valid ground truth for caption semantic alignment.
    Invoked throughout Section 2 and 3.2; the paper uses MOS as the benchmark against which metric correlations are judged.
  • ad hoc to paper The 'paraphrased' captions preserve the semantics of the original captions and the 'distorted' captions change semantics.
    Stated in Section 2 and 3.1; no human ratings are collected for distorted captions (Table 1 leaves MOS blank), so semantic equivalence is asserted rather than verified.
  • domain assumption The sampled subset of MusicCaps clips is representative of the evaluation set.
    The paper randomly samples clips but does not report statistics; Table 2's 20-entry subset is unexplained.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Do Captioning Metrics Reflect Music Semantic Alignment?." pith.science (2026). https://pith.science/paper/NX7G4B2X

@misc{pith2026241111692,
  author       = {Pith},
  title        = {Pith review of: Do Captioning Metrics Reflect Music Semantic Alignment?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NX7G4B2X}},
  note         = {Machine review of arXiv:2411.11692}
}
read the original abstract

Music captioning has emerged as a promising task, fueled by the advent of advanced language generation models. However, the evaluation of music captioning relies heavily on traditional metrics such as BLEU, METEOR, and ROUGE which were developed for other domains, without proper justification for their use in this new field. We present cases where traditional metrics are vulnerable to syntactic changes, and show they do not correlate well with human judgments. By addressing these issues, we aim to emphasize the need for a critical reevaluation of how music captions are assessed.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Assessing Factual Music Comprehension in Large Audio Language Models

    cs.SD 2025-11 conditional novelty 6.0 of 10

    Standard NLP metrics fail to capture factual music understanding in audio-language models; a CLAP-based metric and an LLM-parsed factual QA protocol measure it more directly.

Reference graph

Works this paper leans on

15 extracted references · 11 canonical work pages · cited by 1 Pith paper

  1. [1]

    Do Captioning Metrics reflect Mu- sic Semantic Alignment?

    INTRODUCTION The advancement of music information retrieval (MIR) parallels significant developments in the music industry, particularly regarding tasks like music captioning [1–5]. Despite the promise of music captioning, the evaluation of generated captions poses significant challenges. Current metrics, borrowed from natural language processing tasks, d...

    work page 2024

  2. [2]

    We use Amazon Mechanical Turk [9] to recruit 50 par- ticipants for a listening test

    METHOD To evaluate the effectiveness of traditional metrics in music captioning, we conduct a human evaluation study aimed at correlating human judgments with the scores generated by these metrics. We use Amazon Mechanical Turk [9] to recruit 50 par- ticipants for a listening test. From the evaluation set of the MusicCaps [10] dataset, we randomly sample ...

    work page

  3. [3]

    Do Captioning Metrics Reflect Music Semantic Alignment?

    RESULTS 3.1 Syntactic & Semantic variations Following the collection of human evaluation scores, we report the scores of evaluation metrics (including the Mean Opinion Score (MOS)) for original and paraphrased cap- tions, as shown in Figure 1. Notably, evaluation metrics except for FENSE show a significant decrease when com- paring the original captions t...

    work page Pith review arXiv 2024

  4. [4]

    Given these findings, we conclude that a more nuanced evaluation framework is necessary to ad- dress these challenges

    CONCLUSION We demonstrate existing metrics are overly sensitive to syntactic variations, and they lack alignment with actual human evaluations. Given these findings, we conclude that a more nuanced evaluation framework is necessary to ad- dress these challenges

    work page

  5. [5]

    Muscaps: Generating captions for music audio,

    I. Manco, E. Benetos, E. Quinton, and G. Fazekas, “Muscaps: Generating captions for music audio,” in 2021 International Joint Conference on Neural Net- works (IJCNN). IEEE, 2021, pp. 1–8

    work page 2021

  6. [6]

    Lp-musiccaps: Llm-based pseudo music captioning,

    S. Doh, K. Choi, J. Lee, and J. Nam, “Lp-musiccaps: Llm-based pseudo music captioning,” arXiv preprint arXiv:2307.16372, 2023

    arXiv 2023

  7. [7]

    Llark: A multimodal instruction-following language model for music,

    J. P. Gardner, S. Durand, D. Stoller, and R. M. Bittner, “Llark: A multimodal instruction-following language model for music,” in Forty-first International Confer- ence on Machine Learning, 2023

    work page 2023

  8. [8]

    Mert: Acoustic music understanding model with large-scale self-supervised training,

    Y . Li, R. Yuan, G. Zhang, Y . Ma, X. Chen, H. Yin, C. Lin, A. Ragni, E. Benetos, N. Gyenge et al. , “Mert: Acoustic music understanding model with large-scale self-supervised training,” arXiv preprint arXiv:2306.00107, 2023

    arXiv 2023

Show all 15 references
  1. [9]

    Enriching music descriptions with a finetuned-llm and metadata for text-to-music retrieval,

    S. Doh, M. Lee, D. Jeong, and J. Nam, “Enriching music descriptions with a finetuned-llm and metadata for text-to-music retrieval,” in ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2024, pp. 826–830

    2024

  2. [10]

    Bleu: a method for automatic evaluation of machine transla- tion,

    K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “Bleu: a method for automatic evaluation of machine transla- tion,” in Proceedings of the 40th annual meeting of the Association for Computational Linguistics , 2002, pp. 311–318

    2002

  3. [11]

    Meteor: An automatic met- ric for mt evaluation with improved correlation with human judgments,

    S. Banerjee and A. Lavie, “Meteor: An automatic met- ric for mt evaluation with improved correlation with human judgments,” inProceedings of the acl workshop on intrinsic and extrinsic evaluation measures for ma- chine translation and/or summarization, 2005, pp. 65– 72

    2005

  4. [12]

    Rouge: A package for automatic evalua- tion of summaries,

    C.-Y . Lin, “Rouge: A package for automatic evalua- tion of summaries,” in Text summarization branches out, 2004, pp. 74–81

    2004

  5. [13]

    Amazon mechanical turk,

    “Amazon mechanical turk,” https://www.mturk.com/, accessed: 2024-10-05

    2024

  6. [14]

    Musiclm: Generating music from text,

    A. Agostinelli, T. I. Denk, Z. Borsos, J. Engel, M. Verzetti, A. Caillon, Q. Huang, A. Jansen, A. Roberts, M. Tagliasacchi et al. , “Musiclm: Generating music from text,” arXiv preprint arXiv:2301.11325, 2023

    2023 arXiv

  7. [15]

    Can audio captions be evaluated with image caption metrics?

    Z. Zhou, Z. Zhang, X. Xu, Z. Xie, M. Wu, and K. Q. Zhu, “Can audio captions be evaluated with image caption metrics?” in ICASSP 2022-2022 IEEE Inter- national Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022, pp. 981–985

    2022

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.