Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

From Joy to Fear: A Benchmark of Emotion Estimation in Pop Song Lyrics

T0 review · 3 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that fine-tuning BERT-based models on a new MOS-labeled lyric corpus reduces emotion-score prediction error by more than 30% compared with zero-shot Grok 3.

desk verdict New MOS-labeled lyrics dataset is the only solid contribution; the evaluation tables are internally inconsistent and the central claim is unsupported. read the letter →

arxiv 2509.05617 v1 pith:OTVZCMLE submitted 2025-09-06 cs.CL cs.AI

Shay Dahary , Avi Edana , Alexander Apartsin , Yehudit Aperstein This is my paper · ORCID
classification cs.CLcs.AI
keywords multi-labelemotionclassificationsonglyricsmeanopinionscoreBERTfine-tuningzero-shotLLMEkmanemotionsaffectivecomputingmusicinformationretrieval
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper builds a benchmark for estimating emotional intensity in pop song lyrics. Human raters scored each lyric from 0 to 5 on six emotions, and the average of four raters became the ground-truth label. The authors then compare zero-shot prompting of a large language model with fine-tuned BERT and RoBERTa regression models. They report that both fine-tuned models cut error by more than 30% relative to the zero-shot model, with consistent gains across all six emotions. The paper's conclusion is that when labeled data is available, task-specific fine-tuning beats zero-shot prompting for continuous lyric-emotion prediction.

What carries the argument

The central machinery is the MOS label: each lyric receives a continuous 0 to 5 intensity score per emotion as the arithmetic mean of four independent human ratings, converting subjective annotation into a regression target. On top of this, the paper uses a fine-tuned BERT or RoBERTa encoder with a six-output regression head trained with mean squared error, compared against a zero-shot LLM prompted to return the six numeric scores. The MOS target is what makes the fine-tuning comparison possible, and the regression setup is what lets the paper claim continuous emotion-intensity prediction rather than discrete classification.

What would settle it

Re-run the same comparison on a held-out 20% of the released MOS-labeled dataset and on fresh lyrics never seen during fine-tuning, reporting per-emotion MAE and RMSE. If the fine-tuned BERT and RoBERTa models do not beat zero-shot Grok 3 by roughly 30% on unseen lyrics, the headline claim fails. A second check is to re-label a sample with a different four-rater committee; if the MOS scores change substantially, the ground truth itself is unstable.

Watch

Extended reading notes

Core claim

The paper constructs a manually labeled dataset of pop song lyrics annotated by four raters per lyric on six Ekman emotions—joy, sadness, anger, fear, surprise, and disgust—averaged into continuous mean-opinion scores from 0 to 5. Using that dataset, it evaluates zero-shot prompting of Grok 3 and fine-tuned regression heads on BERT and RoBERTa. The aggregated results show both fine-tuned models reaching MAE 0.33 and RMSE about 0.151, versus MAE 0.50 and RMSE 0.367 for zero-shot Grok, which the paper states is a reduction in error of more than 30%. The improvement is reported across all six emotions, with the smallest MAEs for surprise (0.10) and fear (0.13). The paper concludes that fine-tun

Load-bearing premise

The fine-tuned models are described only as 'trained and validated on the annotated dataset', with no explicit train/test split, cross-validation, or held-out lyrics; if the reported errors come from the same examples used for fitting, the central performance gap is inflated by overfitting.

Editorial extensions

If this is right

  • Task-specific fine-tuning on MOS-labeled lyrics yields materially better continuous emotion-intensity estimates than zero-shot prompting of a frontier LLM, with error reductions above 30% in aggregate MAE and RMSE.
  • The released dataset gives the field a shared benchmark with continuous ground-truth scores for six Ekman emotions across pop lyrics, enabling direct comparisons of future models.
  • The fine-tuned advantage is not confined to one feeling: it holds for joy, sadness, anger, fear, surprise, and disgust.
  • Zero-shot prompting remains a usable fallback when labeled data are unavailable, but the paper positions it as inferior for affective computing and music information retrieval applications.
  • The paper's result extends the fine-tuned-versus-LLM comparison to lyric emotion, aligning with prior evidence that in-domain fine-tuning beats transferring models trained on unrelated text.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not report a train/test split; if the reported errors are in-sample, the central gap is likely overstated. My inference is that a held-out evaluation is the first thing to check before relying on the 30% figure.
  • The zero-shot comparison uses a single prompt template and a single frontier model, so it may understate what LLMs can do with structured output or chain-of-thought prompting.
  • With only four annotators per lyric and no reported inter-annotator agreement, MOS labels may be noisy; a useful extension would compare these labels with Best-Worst Scaling or more raters on the same lyrics.
  • If the gap survives held-out evaluation, the practical consequence is that lightweight BERT-size models can power emotion-aware lyric tagging cheaply, which the paper leaves implicit.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces a manually annotated dataset of pop song lyrics with Mean Opinion Score (MOS) labels for six Ekman emotions (joy, sadness, anger, fear, surprise, disgust) and evaluates both fine-tuned BERT/RoBERTa models and the zero-shot LLM Grok 3 on the task of predicting the six emotion intensities. The headline claim is that fine-tuning BERT-based models reduces error by more than 30% relative to zero-shot Grok, and that these fine-tuned models are preferable for lyric emotion attribution when labeled data are available. The dataset is released publicly.

Significance. A public MOS-labeled lyrics emotion dataset would be a useful resource for affective computing and music information retrieval, and a clean comparison of fine-tuned versus zero-shot LLMs on such a task would inform practical model selection. However, as presented, the quantitative results are internally inconsistent and the evaluation protocol is not described with sufficient detail to support the central claim. The contribution therefore cannot be accepted on the evidence currently in the manuscript.

major comments (3)
  1. [Table II vs. Table III] The aggregated metrics in Table II are internally impossible and contradict Table III. For any set of nonnegative errors, RMSE >= MAE, yet Table II reports fine-tuned BERT MAE=0.33 / RMSE=0.1507 and fine-tuned RoBERTa MAE=0.33 / RMSE=0.1511. Averaging the per-emotion values in Table III for fine-tuned RoBERTa gives MAE ≈ 0.145 and RMSE ≈ 0.33, the reverse pattern. The zero-shot Grok row (MAE=0.50, RMSE=0.367) also disagrees with the per-emotion means in Table III (MAE ≈ 0.40, RMSE ≈ 0.51). Since Section IV's 'more than 30%' improvement claim is derived from Table II, the central quantitative comparison is unsupported as reported.
  2. [Section III.B.1] The fine-tuning procedure is described only as 'trained and validated on the annotated dataset'. No train/test split, cross-validation, or held-out lyrics are described. If the reported errors are computed on the same examples used for fitting, they do not estimate generalization, and the fine-tuned versus zero-shot comparison is confounded by potential train/evaluation overlap. This is a load-bearing methodological omission: the main conclusion depends on the fine-tuned errors being generalization errors.
  3. [Section V] The conclusion states that 'Among the zero-shot models, DeepSeek-R1 demonstrated stronger generalization capabilities', but no DeepSeek-R1 results appear in Tables II or III or anywhere in Section IV. The claimed multi-LLM comparison is thus not actually shown in the manuscript. This missing evidence directly affects the conclusion's breadth.
minor comments (5)
  1. [Section III.A] The number of songs/lyrics in the dataset is never stated. Table I shows four annotators per item, but the text only says 'a diverse committee'. The dataset size, annotation scale, and inter-annotator agreement (e.g., variance or Krippendorff's alpha) should be reported to establish the reliability of the MOS labels.
  2. [Section III.B.2] The zero-shot section mentions 'models' in plural but only names Grok 3. The prompt template used for zero-shot evaluation is not reproduced, which hinders reproducibility, especially because prompt phrasing can strongly affect LLM outputs.
  3. [General] Fine-tuning hyperparameters (learning rate, epochs, batch size, validation split) are not reported. For a benchmark paper, these details are necessary for others to reproduce or build on the results.
  4. [Table III] The Disgust row for zero-shot Grok reads '0.45/057' — likely a typo for '0.45/0.57'. This should be corrected.
  5. [References] Some references appear incomplete or inconsistent (e.g., 'Huang et al. (2021)' is mentioned in the literature review but is not in the reference list). The reference list also mixes conference papers and arXiv preprints without full publication details.

Circularity Check

1 steps flagged · score 6.0 of 10

The fine-tuned models' >30% advantage is computed on the same MOS-labeled dataset used for training, with no held-out split described; the reported 'prediction' errors are fit statistics.

  1. fitted input called prediction [Section III.B.1 and Section IV, Table II/III]
    "The model was trained and validated on the annotated dataset to optimize its predictive accuracy across all six emotions. ... As shown in Table II, both fine-tuned models ... achieved substantially lower MAE and RMSE values compared to the zero-shot Grok LLM, underscoring the effectiveness of task-specific training on our manually annotated dataset. ... fine-tuned models reduce error rates by more than 30% relative to the zero-shot approach."

    The headline claim that fine-tuned models reduce error rates by more than 30% versus zero-shot Grok is quantified by the fine-tuned MAE/RMSE in Table II. But the paper never describes a train/test split, cross-validation, or held-out lyrics; the only evaluation statement is that the models were 'trained and validated on the annotated dataset.' Since the fine-tuned models were optimized to minimize MSE on these same MOS scores, the reported errors are training/validation diagnostics, not out-of-sample predictions, unless an undisclosed split exists. The zero-shot Grok evaluation, by contrast, uses prompts on raw lyrics and never sees the labels. Comparing a fit error to a zero-shot error makes a large gap expected by construction, so the central claim reduces to a fitted input renamed as a

full rationale

The core circularity is the undisclosed evaluation protocol: no test split is mentioned, so the fine-tuned models' errors may be computed on the very MOS labels used for fitting. This makes the >30% improvement over zero-shot Grok partly forced rather than independently demonstrated. Separately, Table II is internally inconsistent: RMSE (0.1507/0.1511) is less than MAE (0.33) for both fine-tuned models, which is impossible for nonnegative errors, and Table III's per-emotion values reverse the relationship (average MAE ~0.145, RMSE ~0.33). This suggests the numbers are swapped or computed on different subsets, further undermining the reported comparison; the conclusion's mention of DeepSeek-R1 is also not backed by any table. I found no self-citation, ansatz-smuggling, uniqueness-import, or renaming circularity. The central claim is therefore unsupported as reported, and to the extent the fine-tuned evaluation uses the training data, it is a fitted input called a prediction.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central contribution is an empirical benchmark, so the ledger is dominated by modeling choices and labeling assumptions rather than derived constants. No new theoretical entities are introduced.

free parameters (2)
  • Zero-shot prompt template = not specified
    The exact wording and output format of the prompt asking Grok for six emotion scores is not shown; it is hand-chosen and directly affects the zero-shot scores.
  • Fine-tuning hyperparameters = not specified
    Learning rate, batch size, epochs, and validation split for BERT/RoBERTa are not reported, so the reported errors cannot be reproduced or compared.
assumptions (4)
  • domain assumption Ekman's six basic emotions are a sufficient taxonomy for pop lyrics
    The paper adopts Ekman (1992) without discussion of whether these six categories capture lyrical affect; other taxonomies (Plutchik) exist in the cited literature.
  • domain assumption MOS from four annotators constitutes reliable ground truth
    Section III.A aggregates four raters with no inter-annotator agreement metric, so label reliability is assumed.
  • domain assumption The reported metrics are computed on data not used for training
    No test split is described in Section III.B; the conclusion depends on the unstated assumption that the fine-tuned models were evaluated out-of-sample.
  • domain assumption The GitHub repository contains the described dataset
    Dataset availability is claimed in the abstract, but no repository contents, size, or license are described in the manuscript.

how reviews work

0 comments
Cite this review

Pith. "Pith review of From Joy to Fear: A Benchmark of Emotion Estimation in Pop Song Lyrics." pith.science (2026). https://pith.science/paper/OTVZCMLE

@misc{pith2026250905617,
  author       = {Pith},
  title        = {Pith review of: From Joy to Fear: A Benchmark of Emotion Estimation in Pop Song Lyrics},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OTVZCMLE}},
  note         = {Machine review of arXiv:2509.05617}
}
read the original abstract

The emotional content of song lyrics plays a pivotal role in shaping listener experiences and influencing musical preferences. This paper investigates the task of multi-label emotional attribution of song lyrics by predicting six emotional intensity scores corresponding to six fundamental emotions. A manually labeled dataset is constructed using a mean opinion score (MOS) approach, which aggregates annotations from multiple human raters to ensure reliable ground-truth labels. Leveraging this dataset, we conduct a comprehensive evaluation of several publicly available large language models (LLMs) under zero-shot scenarios. Additionally, we fine-tune a BERT-based model specifically for predicting multi-label emotion scores. Experimental results reveal the relative strengths and limitations of zero-shot and fine-tuned models in capturing the nuanced emotional content of lyrics. Our findings highlight the potential of LLMs for emotion recognition in creative texts, providing insights into model selection strategies for emotion-based music information retrieval applications. The labeled dataset is available at https://github.com/LLM-HITCS25S/LyricsEmotionAttribution.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Clinical Communication Processing with Models Trained on LLM-Generated Synthetic Data: A Structured Survey and Novel Application Case Studies

    cs.CL 2026-08 conditional novelty 6.0 of 10

    Synthetic clinical communication generated by LLMs can train clinical NLP models in thirteen case studies, but only one is tested on real patient text, leaving transfer to authentic communication unproven.

Reference graph

Works this paper leans on

32 extracted references · 31 canonical work pages · cited by 1 Pith paper

  1. [1]

    The model architecture consisted of the standard BERT encoder followed by a fully connected regression head with six output nodes, one for each emotion

    Fine-Tuned BERT -Based Models : We fine -tuned a pre - trained BERT model for regression -based multi -label prediction, directly predicting the six continuous emotion scores for each lyric. The model architecture consisted of the standard BERT encoder followed by a fully connected regression head with six output nodes, one for each emotion. The loss func...

    work page

  2. [2]

    For zero -shot evaluation, each model was prompted using carefully crafted instructions to predict the emotional intensity scores directly from the raw lyrics

    Zero-Shot Pretrained Models : In addition to fine -tuned models, we investigated the zero -shot emotion classification capabilities of large language models without requiring further task -specific training using the Grok 3 model. For zero -shot evaluation, each model was prompted using carefully crafted instructions to predict the emotional intensity sco...

    work page

  3. [3]

    Baziotis, C., Pelekis, N., & Doulkeridis, C. (2018). Datastories at SemEval -2018 Task 1: Predicting emotion intensity in tweets with deep attentive RNN. In Proceedings of SemEval-2018 (pp. 39–48)

    work page 2018

  4. [4]

    Building high-quality emotion datasets requires reconciling subjective differences in annotator judgments

    Current research suggests that while LLMs can achieve competitive results with effective prompting strat egies, fine - tuned task-specific models still offer superior performance when labeled data is available. Building high-quality emotion datasets requires reconciling subjective differences in annotator judgments. The most common strategy for categorica...

    work page 2018

  5. [5]

    Ameer, I., Bölücü, N., Siddiqui, M. H. F., Can, B., Sidorov, G., & Gelbukh, A. (2023). Multi -label emotion classification in texts using transfer learning. Expert Systems with Applications, 213, 118534

    work page 2023

  6. [6]

    Bareiß, P., Klinger, R., & Barnes, J. (2024). English prompts are better for NLI -based zero -shot emotion classification than target-language prompts. In WWW ’24: Companion Proceedings of the ACM on Web Conference 2024 (pp. 1318 –1326). Association for Compu ting Machinery

    work page 2024

  7. [7]

    Bhaumik, A., & Strzalkowski, T. (2024). Towards a generative approach for emotion detection and reasoning. ArXiv preprint

    work page 2024

  8. [8]

    Boitel, E., Mohasseb, A., & Haig, E. (2024). A comparative analysis of GPT-3 and BERT models for text- based emotion recognition: Performance, efficiency, and robustness. In Advances in Intelligent Systems and Computing, 1453 (pp. 567–579). Springer

    work page 2024

Show all 32 references
  1. [9]

    Buechel, S., & Hahn, U. (2017). Emotion representation learning: bridging lexical and distributional semantics. In Proceedings of NAACL -HLT 2017 (pp. 578 –587). Association for Computational Linguistics

    2017

  2. [10]

    Chen, Y., & Xiao, Y. (2024). Recent advancement of emotion cognition in large language models. ArXiv preprint

    2024

  3. [11]

    Demszky, D., Movshovitz -Attias, D., Ko, J., Cowen, A., Nemade, G., & McAuley, J. (2020). GoEmotions: A dataset of fine -grained emotions. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (pp. 4040–4054)

    2020

  4. [12]

    -W., Lee, K., & Toutanova, K

    Devlin, J., Chang, M. -W., Lee, K., & Toutanova, K. (2019). BERT: Pre -training of deep bidirectional transformers for language understanding. In Proceedings of NAACL -HLT 2019 (pp. 4171 –4186). Association for Computational Linguistics

    2019

  5. [13]

    Edmonds, D., & Sedoc, J. (2021). Multi -emotion classification for song lyrics. In Proceedings of the 11th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis (WASSA 2021) (pp. 221–235). Association for Computational Linguistics

    2021

  6. [14]

    Ekman, P. (1992). An argument for basic emotions. Cognition & Emotion, 6(3–4), 169–200

    1992

  7. [15]

    Felbo, L., Mislove, A., Søgaard, A., Rahwan, I., & Lehmann, S. (2017). Using millions of emoji occurrences to learn any -domain representations for detecting sentiment, emotion, and sarcasm. In Proceedings of the 11th International AAAI Conference on Web an d Social Media (ICW...

    2017

  8. [16]

    Z., Khalid, H

    Khattak, A., Asghar, M. Z., Khalid, H. A., & Ahmad, H. (2022). Emotion classification in poetry text using deep neural network. Multimedia Tools and Applications, 81, 26223–26244

    2022

  9. [17]

    Kim, Y. (2014). Convolutional neural networks for sentence classification. In Proceedings of EMNLP 2014 (pp. 1746–1751)

    2014

  10. [18]

    Li, C., Wang, J., Zhang, Y., Zhu, K., Hou, W., Lian, J., Luo, F., Yang, Q., & Xie, X. (2023). Large language models understand and can be enhanced by emotional stimuli—ArXiv preprint

    2023

  11. [19]

    Liu, X., Shi, T., Zhou, G., Liu, M., Yin, Z., Yin, L., & Zheng, W. (2023). Emotion classification for short texts: an improved multi -label method. Humanities and Social Sciences Communications, 10, Article 306

    2023

  12. [20]

    & Stoyanov, V

    Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., ... & Stoyanov, V. (2019). RoBERTa: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692

    2019 arXiv

  13. [21]

    Liu, Z., Yang, K., Xie, Q., Zhang, T., & Ananiadou, S. (2024). EmoLLMs: A series of emotional large language models and annotation tools for comprehensive affective analysis. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD, 2024)

    2024

  14. [22]

    Mihalcea, R., & Strapparava, C. (2012). Lyrics, music, and emotions. In Proceedings of EMNLP 2012 (pp. 590–599)

    2012

  15. [23]

    M., Bravo -Marquez, F., Salameh, M., & Kiritchenko, S

    Mohammad, S. M., Bravo -Marquez, F., Salameh, M., & Kiritchenko, S. (2018). SemEval -2018 Task 1: Affect in Tweets. In Proceedings of the 12th International Workshop on Semantic Evaluation (pp. 1 –17). Association for Computational Linguistics

    2018

  16. [24]

    Peng, L., Zhang, Z., Pang, T., Han, J., & Xiao, Y. (2023). Improving emotion recognition with LLMs: Adaptation of large language models for emotion recognition. ArXiv preprint

    2023

  17. [25]

    & Liu, P

    Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., ... & Liu, P. J. (2020). Exploring the limits of transfer learning with a unified text -to-text transformer. Journal of Machine Learning Research, 21(140), 1–67

    2020

  18. [26]

    Song, Y., & Beck, D. (2023). Modeling emotion dynamics in song lyrics with state space models. Transactions of the Association for Computational Linguistics, 11, 157–175

    2023

  19. [27]

    S., & Mahalakshmi, G

    Sreeja, P. S., & Mahalakshmi, G. S. (2019). Emotion recognition in poetry using an ensemble of classifiers. In Next Generation Computing Technologies on Computational Intelligence (pp. 77 –91). Springer. (Communications in Computer and Information Science, 922)

    2019

  20. [28]

    Strapparava, C., Mihalcea, R., & Battocchi, A. (2012). A parallel corpus of music and lyrics annotated with emotions. In Proceedings of LREC 2012 (pp. 2343–2346)

    2012

  21. [29]

    N., & Gratch, J

    Tak, A. N., & Gratch, J. (2024). GPT -4 emulates average- human emotional cognition from a third -person perspective—ArXiv preprint

    2024

  22. [30]

    Venkatakrishnan, R., Goodarzi, M., & Canbaz, M. A. (2023). Exploring large language models’ emotion detection abilities: Use cases from the Middle East. In Proceedings of the 2023 IEEE Conference on Artificial Intelligence (CAI, 2023) (pp. 241–244). IEEE

    2023

  23. [31]

    Wake, N., Kanehira, A., Sasabuchi, K., Takamatsu, J., & Ikeuchi, K. (2023). Bias in emotion recognition with ChatGPT. ArXiv preprint

    2023

  24. [32]

    Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., & Le, Q. V. (2019). XLNet: A Generalized Autoregressive Pretraining Approach for Language Understanding. In Advances in Neural Information Processing Systems, 32 (pp. 5753–5763)

    2019

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.