Pith. sign in

REVIEW 3 major objections 6 minor 2 cited by

LLM-based music recommenders cannot be judged by accuracy alone; evaluation must move to groundedness, discovery, and risk diagnostics.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 21:05 UTC pith:PTKXT6YT

load-bearing objection A credible, well-organized position paper that gives the MRS community a useful vocabulary for evaluating LLM-based recommenders, but its proposed new metrics are mostly sketches awaiting validation. the 3 major comments →

arxiv 2511.16478 v2 pith:PTKXT6YT submitted 2025-11-20 cs.IR cs.CL

Music Recommendation with Large Language Models: Challenges, Opportunities, and Evaluation

Elena V. Epure , Yashar Deldjoo , Bruno Sguerra , Markus Schedl , Manuel Moussallam This is my paper
classification cs.IR cs.CL
keywords Music recommender systemsLarge language modelsEvaluation metricsGroundednessHallucinationPersonalizationNatural language recommendationBeyond-accuracy evaluation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This review argues that LLM-driven music recommender systems break the field's dominant evaluation paradigm, which equates a good recommendation with accurately predicting what users consume. Because LLMs generate text instead of ranking items, standard accuracy metrics miss hallucinations, non-determinism, and personalization, and train/test separation is undermined by opaque training data. The paper proposes a structured alternative: six success dimensions (groundedness, discovery, personalization gain, profile fidelity, cultural/linguistic coverage, classical relevance) plus a complementary set of risk diagnostics, adapted from NLP evaluation practices. If adopted, studies of LLM recommenders would report whether recommendations are real, grounded, diverse, and controllable, not just whether they match logged consumption.

Core claim

The paper's central claim is that evaluating an LLM-based music recommender as if it were a retrieval system—scoring precision and recall against consumption logs—no longer measures whether recommendations are good. LLMs are generative, non-deterministic, and prone to hallucination, so the field must rethink evaluation from the ground up. The paper establishes that success should be assessed along six dimensions: query adherence and groundedness, discovery quality, personalization gain, profile fidelity and controllability, cultural and linguistic coverage, and classical relevance. It then adds a complementary set of risk diagnostics covering hallucination, popularity/temporal/language bias,

What carries the argument

The central object is the paper's proposed evaluation framework: a decision tree for selecting NLP-derived metrics, plus a catalog of six success dimensions (G1–G6) and eight risk diagnostics. For the natural-language recommendation scenario, the task is defined as producing a ranked list of k resolvable catalog items from a prompt that may include a user request, a natural-language profile, in-context examples, retrieved documents, or a reasoning scaffold. The framework's work is to separate faithful grounding from fluent text, and to make hallucination, bias, and instability measurable rather than hidden inside a single accuracy number.

Load-bearing premise

The framework's prescriptions depend on the assumption that the proposed evaluation measures can be reliably computed and validated—especially LLM-based judges, whose reliability for music content is itself an open question the paper flags.

What would settle it

A large-scale user study would settle it: collect natural-language music requests, have an LLM produce recommendation lists, then compare human judgments of quality against both standard accuracy metrics and the paper's groundedness and discovery metrics; if accuracy correlates as strongly with human preference as the proposed metrics do, the claim that accuracy metrics are inadequate collapses.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If the paper is right, accuracy-only benchmarks become insufficient for LLM recommenders; studies would need to report groundedness, discovery, personalization gain, and coverage alongside classical relevance.
  • Entity resolution against a catalog or knowledge base should become a standard evaluation step, so hallucinated tracks and misattributed metadata are penalized explicitly.
  • LLM-as-judge methods should not be adopted in music without first being validated against human judgments, because position, verbosity, and self-enhancement biases can distort scores.
  • Each prompting setting needs specialized tests: shot calibration and prompt sensitivity for in-context learning, evidence grounding and document-swap tests for retrieval-augmented generation, and reasoning faithfulness plus self-consistency for chain-of-thought prompting.
  • Because black-box LLMs may have been exposed to public benchmark datasets during training, offline accuracy results are hard to interpret, and evaluation should be reported with contamination awareness.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, this evaluation framework likely generalizes to other generative recommender domains such as movies, books, and e-commerce, since the core problem—ranking-based metrics failing on generative outputs—is not music-specific.
  • The risk diagnostics could be operationalized into a standardized 'hallucination rate' and 'bias report card' for deployed music assistants, enabling direct comparison across systems in a way that accuracy metrics do not.
  • The emphasis on profile fidelity suggests a testable interactive protocol: let users edit their natural-language profiles, then measure how quickly and consistently the recommender's outputs adapt; such a protocol is not yet standard in offline evaluation.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper argues that LLM-driven music recommender systems (MRS) require a fundamental rethinking of evaluation, because LLMs are generative, non-deterministic, prone to hallucination, and trained on opaque data, which breaks the assumptions behind traditional accuracy/IR metrics. It reviews LLM applications to user modeling, item modeling, and natural-language recommendation in music, surveys NLP evaluation methods (reference-based, reference-free, human/LLM-as-judge) and their risks, and proposes a structured set of six success dimensions (G1–G6) and eight risk dimensions, together with setting-specific bundles for in-context learning, retrieval-augmented generation, and chain-/tree-of-thought prompting. The paper is explicitly a position/perspective piece: it provides no empirical validation of the proposed metrics, but it is transparent about open questions, especially the reliability of LLM-as-judge evaluation.

Significance. If the central claim is accepted, the paper is a timely and useful synthesis. It compiles a wide range of MRS-specific challenges and connects them to evaluation practices from NLP, going beyond a simple accuracy-oriented framing. The structured catalog of success and risk dimensions provides a concrete reference point for future work, and the separation of catalog, resolver/knowledge base, and prompting components is a helpful modeling choice. Strengths include the explicit treatment of non-determinism and decoding variability, the recognition that LLM-as-judge must be validated before adoption, and the inclusion of risk dimensions such as popularity/temporal/language bias and sampling-bias amplification. As a position paper, the lack of empirical validation is not fatal, but the informal status of several proposed metrics should be clarified so that readers do not mistake a research agenda for a validated framework.

major comments (3)
  1. [3.2.1] The proposed success metrics for S-RAG and S-COT — EvidenceGrounding@k, Δdoc, TTFS, Day-0 Lift, ΔU_CoT — are introduced with informal hedges ('one might define', 'one might introduce') and no formal definitions, aggregation rules, or validation protocol. Since the paper's constructive contribution is precisely this catalog, the prescriptive force of G1–G6 depends on these metrics being operationalizable. Please either (a) provide precise definitions (pseudo-code or formulas) with a validation plan against human judgment, or (b) explicitly label Section 3.2 as a research agenda and add a paragraph on the validation steps needed. As written, the reader cannot tell which parts of the framework are intended as standards and which as conjectures.
  2. [3.1.3, 3.2.2(4)] The paper correctly states in Section 3.1.3 that LLM-as-judge must be rigorously validated against human judgments before adoption, and Section 3.2.2(4) lists evaluator bias as a risk dimension. However, several of the proposed success/risk measures (groundedness, faithfulness, reasoning faithfulness) are described as LLM-based assessments. This creates a circular dependency if the framework itself relies on the very evaluation approach whose reliability it flags as an open problem. Please state explicitly how the circularity is broken: which metrics should be computed only with human annotation, which can use LLM-as-judge after calibration, and what acceptance criteria (e.g., agreement thresholds) are proposed.
  3. [3.2.1–3.2.2] The framework provides separate success (G1–G6) and risk dimensions but no guidance on joint reporting or trade-offs. For example, a system may achieve high G6 accuracy while exhibiting high popularity bias (risk 2), or high G2 discovery at the cost of latency/token budget in S-RAG. A short subsection on how to report these dimensions together (e.g., a required disclosure format, or a minimum-quality threshold on risk dimensions) would make the framework actionable. Without this, the catalog remains a list of dimensions rather than a coherent evaluation protocol.
minor comments (6)
  1. [Figure 4 (Section 3.1)] The diagram appears to contain a stray browser/export header '20/11/2025, 16:37 nlp-eval Page 1 of 2 https://app.diagrams.net/' and the text is rendered as an image rather than typeset text. This will make the figure hard to read and contains an unintended URL/timestamp artifact. Please replace with a clean, typeset figure.
  2. [References] Several references are incomplete: [20] and [161] use 'et al.' with no author names, and [174] contains the placeholder 'arXiv:2502.xxxxx'. Also, [165] and [166] duplicate the same self-consistency work, and [55] lists 'Haofen Wang' twice. Please clean up the reference list.
  3. [Table in 3.2.1] The table's setting 'S-Prompting — In-Context Learning & Chain-of-Thought' merges ICL and CoT, while the text uses separate S-ICL and S-COT subsections. Align the nomenclature to avoid confusion.
  4. [3.2.3] The phrase 'most straightaway methods of evaluating' should be 'most straightforward methods'.
  5. [G4] The citation list in G4 includes the same reference twice as [151,151]; check and deduplicate.
  6. [Introduction/Abstract] The paper sometimes says 'accuracy-only metrics are questionable' but later retains G6 classical relevance as a core dimension. Make explicit that the argument is for supplementation rather than abandonment of accuracy metrics, to avoid misreading.

Circularity Check

0 steps flagged

No significant circularity: the paper is a position/survey proposing evaluation dimensions, with no derivation that reduces to its inputs.

full rationale

This is a survey/position paper, not a derivation. Its central claim—that LLM-driven MRS requires rethinking evaluation—is argued from external observations about generative models, known NLP evaluation practice, and cited empirical studies. The proposed success and risk dimensions (G1–G6, S-ICL, S-RAG, S-CoT) are explicitly framed as a catalog, with hedged definitions such as 'one might define EvidenceGrounding@k' and 'one might introduce freshness... metrics such as TTFS and Day-0 Lift.' These are proposed metrics, not fitted parameters presented as predictions, and the paper does not claim to have validated them. The self-citations (e.g., Deldjoo et al. [30] on ChatGPT bias, Sguerra et al. [144] on LLM-generated taste profiles) are used as empirical evidence from peer-reviewed venues, not as a self-referential uniqueness theorem or ansatz; they support the motivation for risk dimensions but are not the sole basis for any derived result. The paper itself flags the key open problem that LLM-as-judge must be validated before adoption ('LLMs should be rigorously validated against human judgments before being adopted as automatic evaluators'), which is an acknowledged limitation rather than a circular step. No equation or definition equates an output to an input by construction, and no fitted value is renamed as a prediction. Therefore, no significant circularity is present.

Axiom & Free-Parameter Ledger

0 free parameters · 4 axioms · 0 invented entities

This is a review/position paper: it contains no fitted parameters and postulates no new entities. Its central claim rests on domain assumptions about the recommendation task, the availability of knowledge bases, the applicability of NLP evaluation, and the availability of ground-truth references.

axioms (4)
  • domain assumption The primary evaluation scenario is a ranked list of k resolvable catalog items produced in response to an NL request (possibly conditioned on profile, ICL, RAG, or CoT).
    Section 2.3 defines the task components (catalog, resolver/KB, NL request, profile, in-context examples, RAG docs, reasoning scaffold) and the whole framework in Section 3 is scoped to this scenario.
  • domain assumption External knowledge bases (e.g., MusicBrainz) are available and reliable for entity resolution and grounding.
    Section 2.3 says 'we explicitly separate the catalog... from knowledge bases' and the evaluation relies on KB entity resolution and EntitiesResolvedShare.
  • domain assumption NLP evaluation metrics (reference-based, reference-free, human/LLM-as-judge) are applicable to MRS outputs.
    Section 3.1 adapts NLP evaluation to MRS; the paper itself notes the open question of whether LLM-based evaluators reliably assess music-related text (Section 3.1.3).
  • domain assumption Ground-truth references or human judgments are available or obtainable for evaluation.
    Reference-based evaluation in Section 3.1.1 relies on ground-truth outputs y*; the paper acknowledges human evaluation is costly and often unavailable.

pith-pipeline@v1.3.0-alltime-deepseek · 38740 in / 10016 out tokens · 81222 ms · 2026-08-03T21:05:42.253374+00:00 · methodology

0 comments
read the original abstract

Music Recommender Systems (MRSs) have long relied on an information retrieval framing, where progress is measured mainly through accuracy on retrieval-oriented subtasks. While effective, this reductionist paradigm struggles to address the deeper question of what makes a good recommendation. Attempts to broaden evaluation, through user studies or fairness analyses, have had limited impact. The emergence of Large Language Models (LLMs) disrupts this framework: LLMs are generative rather than ranking-based, making standard accuracy metrics questionable. They also introduce challenges such as hallucinations, knowledge cutoffs, non-determinism, and opaque training data, rendering traditional train or test protocols difficult to interpret. At the same time, LLMs create new opportunities, enabling natural language (NL) interaction and even allowing models to act as evaluators. This work argues that the shift toward LLM-driven MRSs requires rethinking evaluation. We first review how LLMs reshape user modeling, item modeling, and NL-based recommendation in music. We then examine evaluation practices from NLP, highlighting methodologies and open challenges relevant to MRSs. Finally, we synthesize insights, focusing on how LLM prompting applies to MRSs, to outline a structured set of success and risk dimensions. Our goal is to provide the MRSs community with an updated, pedagogical, and cross-disciplinary perspective on evaluation.

Figures

Figures reproduced from arXiv: 2511.16478 by Bruno Sguerra, Elena V. Epure, Manuel Moussallam, Markus Schedl, Yashar Deldjoo.

Figure 1
Figure 1. Figure 1: Paper’s overview as a generic diagram presenting music recommendation with LLMs [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Representation of the method to derive NL user preference profiles from consumption data. [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Prompt example with the different task components highlighted. [PITH_FULL_IMAGE:figures/full_fig_p011_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: NLP-driven evaluation framework: decision process and metrics overview. [PITH_FULL_IMAGE:figures/full_fig_p019_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Music Playlist Captioning at Scale with Large Language Models

    cs.IR 2026-06 unverdicted novelty 4.0

    Deezer deployed an LLM-driven playlist captioning system in 2025 for its Daily Mix recommendations, claiming significant gains in user engagement from the added natural-language descriptions.

  2. MuChator: Enabling Active Music Discovery via Conversational Music LLMs in Douyin Music

    cs.IR 2026-05 unverdicted novelty 4.0

    MuChator introduces a three-component MusicLLM system (staged knowledge pre-training, automated triplet instruction tuning, hybrid RM with GRPO) that outperforms Gemini-3-Pro on internal datasets and yields 46.49% hig...

Reference graph

Works this paper leans on

196 extracted references · 17 canonical work pages · cited by 2 Pith papers

  1. [1]

    Marah I Abdin, Suriya Gunasekar, Varun Chandrasekaran, Jerry Li, Mert Yuksekgonul, Rahee Ghosh Peshawaria, Ranjita Naik, and Besmira Nushi. 2024. KITAB: Evaluating LLMs on Constraint Satisfaction for Information Retrieval. InThe Twelfth International Conference on Learning Representations (ICLR). https://openreview.net/forum?id=b3kDP3IytM

    2024

  2. [2]

    Himan Abdollahpouri, Masoud Mansoury, Robin Burke, and Bamshad Mobasher. 2019. Managing Popularity Bias in Recommender Systems with Personalized Re-ranking. InProceedings of the 29th International Florida Artificial Intelligence Research Society Conference (FLAIRS). 413–418

    2019

  3. [3]

    Khetam Al Sharou, Zhenhao Li, and Lucia Specia. 2021. Towards a Better Understanding of Noise in Natural Language Processing. InProceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021), Ruslan Mitkov and Galia Angelova (Eds.). INCOMA Ltd., Held Online, 53–62. https://aclanthology.org/2021.ranlp-1.7/

    2021

  4. [4]

    Geetha Sai Aluri, Siddharth Sharma, Tarun Sharma, and Joaquin Delgado. 2024. Playlist Search Reinvented: LLMs Behind the Curtain. InProceedings of the 18th ACM Conference on Recommender Systems(Bari, Italy)(RecSys ’24). Association for Computing Machinery, New York, NY, USA, 813–815. doi:10.1145/3640457.3688047

    arXiv 2024

  5. [5]

    Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. 2024. Self-rag: Learning to retrieve, generate, and critique through self-reflection. (2024)

    2024

  6. [6]

    Yuyang Bai, Shangbin Feng, Vidhisha Balachandran, Zhaoxuan Tan, Shiqi Lou, Tianxing He, and Yulia Tsvetkov. 2024. KGQuiz: Evaluating the Generalization of Encoded Knowledge in Large Language Models. InProceedings of the ACM Web Conference 2024(Singapore, Singapore)(WWW ’24). Association for Computing Machinery, New York, NY, USA, 2226–2237. doi:10.1145/35...

    arXiv 2024

  7. [8]

    Keqin Bao, Ming Yan, Yang Zhang, Jizhi Zhang, Wenjie Wang, Fuli Feng, and Xiangnan He. 2024. Real-Time Personalization for LLM-based Recommendation with Customized In-Context Learning.arXiv preprint arXiv:2410.23136(2024)

    Pith/arXiv arXiv 2024

  8. [9]

    Christine Bauer and Markus Schedl. 2019. Global and country-specific mainstreaminess measures: Definitions, analysis, and usage for improving personalized music recommendation systems.PLOS ONE14, 6 (June 2019), e0217389. doi:10.1371/journal.pone.0217389 Publisher: Public Library of Science

    work page doi:10.1371/journal.pone.0217389 2019

  9. [10]

    Anna Bavaresco, Raffaella Bernardi, Leonardo Bertolazzi, Desmond Elliott, Raquel Fernández, Albert Gatt, Esam Ghaleb, Mario Giulianelli, Michael Hanna, Alexander Koller, Andre Martins, Philipp Mondorf, Vera Neplenbroek, Sandro Pezzelle, Barbara Plank, David Schlangen, Alessandro Suglia, Aditya K Surikuchi, Ece Takmaz, and Alberto Testoni. 2025. LLMs inste...

    2025

  10. [11]

    Giovanni Maria Biancofiore, Tommaso Di Noia, Eugenio Di Sciascio, Fedelucio Narducci, and Paolo Pastore. 2022. Aspect based sentiment analysis in music: a case study with spotify. InProceedings of the 37th ACM/SIGAPP Symposium on Applied Computing(Virtual Event)(SAC ’22). Association for Computing Machinery, New York, NY, USA, 696–703. doi:10.1145/3477314.3507092

    arXiv 2022

  11. [12]

    Tony Brooke. 2014. descriptive metadata in the music industry: Why it is broken and how to fix it—part one.Journal of Digital Media Management 2, 3 (2014), 263–282

    2014

  12. [13]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin...

    Pith/arXiv arXiv 2020

  13. [14]

    Pedro G Campos, Fernando Díez, and Iván Cantador. 2014. Time-aware recommender systems: a comprehensive survey and analysis of existing evaluation protocols.User Modeling and User-Adapted Interaction24, 1 (2014), 67–119

    2014

  14. [15]

    Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. 2021. Extracting Training Data from Large Language Models. In30th USENIX Security Symposium (USENIX Security 21). 2633–2650

    2021

  15. [16]

    Brandon James Carone and Pablo Ripollés. 2024. SoundSignature: What Type of Music do you Like?. In2024 IEEE 5th International Symposium on the Internet of Sounds (IS2). IEEE, 1–10

    2024

  16. [17]

    Elif Celen, Pol van Rijn, Harin Lee, and Nori Jacoby. 2025. Are Expressions for Music Emotions the Same Across Cultures?. InProceedings of the Annual Meeting of the Cognitive Science Society (CogSci)

    2025

  17. [18]

    2009.Music recommendation and discovery in the long tail

    Òscar Celma Herrada et al. 2009.Music recommendation and discovery in the long tail. Universitat Pompeu Fabra

    2009

  18. [19]

    Ching-Wei Chen, Paul Lamere, Markus Schedl, and Hamed Zamani. 2018. Recsys challenge 2018: Automatic music playlist continuation. In Proceedings of the 12th ACM Conference on Recommender Systems. 527–528

    2018

  19. [20]

    et al. Chen. 2025. MLA-Trust: Benchmarking Trustworthiness of Multimodal Large-scale Agents.arXiv preprint arXiv:2506.01616(2025)

    Pith/arXiv arXiv 2025

  20. [21]

    Jiawei Chen, Hande Dong, Xiang Wang, Fuli Feng, Meng Wang, and Xiangnan He. 2023. Bias and debias in recommender system: A survey and future directions.ACM Transactions on Information Systems41, 3 (2023), 1–39

    2023

  21. [22]

    Yanran Chen and Steffen Eger. 2023. MENLI: Robust Evaluation Metrics from Natural Language Inference.Transactions of the Association for Computational Linguistics11 (2023), 804–825. doi:10.1162/tacl_a_00576

    doi:10.1162/tacl_a_00576 2023

  22. [23]

    Yi Chen, Rui Wang, Haiyun Jiang, Shuming Shi, and Ruifeng Xu. 2023. Exploring the Use of Large Language Models for Reference-Free Text Quality Evaluation: An Empirical Study. InFindings of the Association for Computational Linguistics: IJCNLP-AACL 2023 (Findings), Jong C. Park, Yuki Arase, Baotian Hu, Wei Lu, Derry Wijaya, Ayu Purwarianti, and Adila Alfa ...

    work page doi:10.18653/v1/2023.findings-ijcnlp.32 2023

  23. [24]

    Cheng-Han Chiang and Hung-yi Lee. 2023. Can Large Language Models Be an Alternative to Human Evaluations?. InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (Eds.). Association for Computational Linguistics, Toronto, Canada, 15607–15631. do...

    doi:10.18653/v1/2023.acl-long.870 2023

  24. [25]

    Keunwoo Choi, Seungheon Doh, and Juhan Nam. 2025. TalkPlayData 2: An Agentic Synthetic Data Pipeline for Multimodal Conversational Music Recommendation.arXiv preprint arXiv:2509.09685(2025)

    Pith/arXiv arXiv 2025

  25. [26]

    Pierre Colombo, Guillaume Staerman, Chloé Clavel, and Pablo Piantanida. 2021. Automatic Text Evaluation through the Lens of Wasserstein Barycenters. InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (Eds.). Association for Computational Lingui...

    work page doi:10.18653/v1/2021.emnlp-main.817 2021

  26. [27]

    Microsoft Corporation. 2025. GroundednessEvaluator — Azure AI Evaluation SDK for Python. https://learn.microsoft.com/en-us/python/api/azure- ai-evaluation/azure.ai.evaluation.groundnessevaluator?view=azure-python. Accessed: 2025-10-25

    2025

  27. [28]

    Paolo Cremonesi and Dietmar Jannach. 2021. Progress in recommender systems research: Crisis? What crisis?AI Magazine42, 3 (2021), 43–54

    2021

  28. [29]

    Mathieu Delcluze, Antoine Khoury, Clémence Vast, Valerio Arnaudo, Léa Briand, Walid Bendada, and Thomas Bouabça. 2025. Text2Playlist: Generating Personalized Playlists from Text on Deezer. InThe 47th European Conference on Information Retrieval (ECIR 2025). Manuscript submitted to ACM Music Recommendation with Large Language Models 29

    2025

  29. [30]

    Yashar Deldjoo. 2024. Understanding biases in chatgpt-based recommender systems: Provider fairness, temporal stability, and recency.ACM Transactions on Recommender Systems(2024)

    2024

  30. [31]

    Yashar Deldjoo, Zhankui He, Julian McAuley, Anton Korikov, Scott Sanner, Arnau Ramisa, Rene Vidal, Maheswaran Sathiamoorthy, Atoosa Kasrizadeh, Silvia Milano, et al. 2024. Recommendation with Generative Models.arXiv preprint arXiv:2409.15173(2024)

    Pith/arXiv arXiv 2024

  31. [32]

    Yashar Deldjoo, Nikhil Mehta, Maheswaran Sathiamoorthy, Shuai Zhang, Pablo Castells, and Julian McAuley. 2025. Toward Holistic Evaluation of Recommender Systems Powered by Generative Models. InProceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval. 3932–3942

    2025

  32. [33]

    Yashar Deldjoo, Markus Schedl, and Peter Knees. 2024. Content-driven music recommendation: Evolution, state of the art, and challenges.Computer Science Review51 (2024), 100618. doi:10.1016/j.cosrev.2024.100618

    arXiv 2024

  33. [34]

    Zihao Deng, Yinghao Ma, Yudong Liu, Rongchen Guo, Ge Zhang, Wenhu Chen, Wenhao Huang, and Emmanouil Benetos. 2024. MusiLingo: Bridging Music and Text with Pre-trained Language Models for Music Captioning and Query Response. InFindings of the Association for Computational Linguistics: NAACL 2024. 3643–3655

    2024

  34. [35]

    Daniel Deutsch, Rotem Dror, and Dan Roth. 2022. On the Limitations of Reference-Free Evaluations of Generated Text. InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (Eds.). Association for Computational Linguistics, Abu Dhabi, United Arab Emirates, 10960–10977. doi:10...

    doi:10.18653/v1/2022.emnlp-main.753 2022

  35. [36]

    Karlijn Dinnissen and Christine Bauer. 2022. Fairness in music recommender systems: A stakeholder-centered mini review.Frontiers in big Data5 (2022), 913608

    2022

  36. [37]

    Seungheon Doh, Keunwoo Choi, Jongpil Lee, and Juhan Nam. 2023. LP-MusicCaps: LLM-Based Pseudo Music Captioning. InIsmir 2023 Hybrid Conference

    2023

  37. [38]

    Seungheon Doh, Junwon Lee, and Juhan Nam. 2021. Music Playlist Title Generation: A Machine-Translation Approach. InProceedings of the 2nd Workshop on NLP for Music and Spoken Audio (NLP4MusA), Sergio Oramas, Elena Epure, Luis Espinosa-Anke, Rosie Jones, Massimo Quadrana, Mohamed Sordo, and Kento Watanabe (Eds.). Association for Computational Linguistics, ...

    2021

  38. [39]

    SeungHeon Doh, Minhee Lee, Dasaem Jeong, and Juhan Nam. 2024. Enriching music descriptions with a finetuned-llm and metadata for text-to-music retrieval. InICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 826–830

    2024

  39. [40]

    Elena Epure and Romain Hennequin. 2023. A human subject study of named entity recognition in conversational music recommendation queries. InProceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics. 1281–1296

    2023

  40. [41]

    Epure, Anis Khlif, and Romain Hennequin

    Elena V. Epure, Anis Khlif, and Romain Hennequin. 2019. Leveraging knowledge bases and parallel annotations for music genre translation. In International Society for Music Information Retrieval Conference

    2019

  41. [42]

    Elena V Epure, Guillaume Salha, and Romain Hennequin. 2020. Multilingual music genre embeddings for effective cross-lingual music item annotation.arXiv preprint arXiv:2009.07755(2020)

    Pith/arXiv arXiv 2020

  42. [43]

    Elena V Epure, Guillaume Salha, Manuel Moussallam, and Romain Hennequin. 2020. Modeling the Music Genre Perception across Language-Bound Cultures. InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). 4765–4779

    2020

  43. [44]

    Shahul Es, Jithin James, Luis Espinosa-Anke, and Steven Schockaert. 2023. RAGAS: Automated Evaluation of Retrieval Augmented Generation. arXiv preprint arXiv:2309.15217(2023)

    Pith/arXiv arXiv 2023

  44. [45]

    Shahul Es, Jithin James, Luis Espinosa Anke, and Steven Schockaert. 2024. RAGAs: Automated Evaluation of Retrieval Augmented Generation. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, Nikolaos Aletras and Orphee De Clercq (Eds.). Association for Computational Linguisti...

    doi:10.18653/v1/2024.eacl-demo.16 2024

  45. [46]

    Andres Ferraro. 2019. Music cold-start and long-tail recommendation: bias in deep representations. InProceedings of the 13th ACM conference on recommender systems. 586–590

    2019

  46. [47]

    Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, George Foster, Alon Lavie, and Ondřej Bojar. 2021. Results of the WMT21 Metrics Shared Task: Evaluating Metrics with Expert-based Human Evaluations on TED and News Domain. InProceedings of the Sixth Conference on Machine Translation, Loic Barrault, Ondrej Bojar, Fethi Bougares, Rajen C...

    2021

  47. [48]

    Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu. 2024. GPTScore: Evaluate as You Desire. InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), Kevin Duh, Helena Gomez, and Steven Bethard (Eds.). Association for Computational Linguis...

    doi:10.18653/v1/2024.naacl-long.365 2024

  48. [49]

    Yumeng Fu, Junjie Wu, Zhongjie Wang, Meishan Zhang, Lili Shan, Yulin Wu, and Bingquan Liu. 2025. LaERC-S: Improving LLM-based Emotion Recognition in Conversation with Speaker Characteristics. InProceedings of the 31st International Conference on Computational Linguistics, Owen Rambow, Leo Wanner, Marianna Apidianaki, Hend Al-Khalifa, Barbara Di Eugenio, a...

    2025

  49. [50]

    Christian Fuentes, Johan Hagberg, and Hans Kjellberg. 2019. Soundtracking: music listening practices in the digital age.European Journal of Marketing53, 3 (2019), 483–503

    2019

  50. [51]

    Escrig, and M

    Nieves Fuentes-Sánchez, Raúl Pastor, Tuomas Eerola, Miguel A. Escrig, and M. Carmen Pastor. 2022. Musical preference but not familiarity influences subjective ratings and psychophysiological correlates of music-induced emotions.Personality and Individual Differences198 (Nov. 2022), Manuscript submitted to ACM 30 Epure et al. 111828. doi:10.1016/j.paid.2022.111828

    arXiv 2022

  51. [52]

    Giovanni Gabbolini and Derek Bridge. 2021. Generating Interesting Song-to-Song Segues With Dave. InProceedings of the 29th ACM Conference on User Modeling, Adaptation and Personalization(Utrecht, Netherlands)(UMAP ’21). Association for Computing Machinery, New York, NY, USA, 98–107. doi:10.1145/3450613.3456819

    arXiv 2021

  52. [53]

    Giovanni Gabbolini, Romain Hennequin, and Elena Epure. 2022. Data-Efficient Playlist Captioning With Musical and Linguistic Knowledge. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (Eds.). Association for Computational Linguistics, Abu Dhabi, United Arab Emirates,...

    work page doi:10.18653/v1/2022.emnlp-main.784 2022

  53. [54]

    Qingqing Gao, Jiuxin Cao, Biwei Cao, Xin Guan, and Bo Liu. 2024. CEPT: A Contrast-Enhanced Prompt-Tuning Framework for Emotion Recognition in Conversation. InProceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC- COLING 2024), Nicoletta Calzolari, Min-Yen Kan, Veronique Hoste, Alessand...

    2024

  54. [55]

    Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yixin Dai, Jiawei Sun, Haofen Wang, and Haofen Wang. 2023. Retrieval- augmented generation for large language models: A survey.arXiv preprint arXiv:2312.109972, 1 (2023)

    Pith/arXiv arXiv 2023

  55. [56]

    Zhaolin Gao, Joyce Zhou, Yijia Dai, and Thorsten Joachims. 2024. End-to-end Training for Recommendation with Language-based User Profiles. arXiv preprint arXiv:2410.18870(2024)

    Pith/arXiv arXiv 2024

  56. [57]

    Iker García-Ferrero, Begoña Altuna, Javier Alvez, Itziar Gonzalez-Dios, and German Rigau. 2023. This is not a Dataset: A Large Negation Benchmark to Challenge Large Language Models. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Association for Computational Lingu...

    doi:10.18653/v1/2023.emnlp-main.531 2023

  57. [58]

    Josh Gardner, Simon Durand, Daniel Stoller, and Rachel Bittner. 2024. LLARK: a multimodal instruction-following language model for music. In Proceedings of the 41st International Conference on Machine Learning(Vienna, Austria)(ICML’24). JMLR.org, Article 603, 46 pages

    2024

  58. [59]

    Tao Ge, Xin Chan, Xiaoyang Wang, Dian Yu, Haitao Mi, and Dong Yu. 2025. Scaling Synthetic Data Creation with 1,000,000,000 Personas. doi:10.48550/arXiv.2406.20094 arXiv:2406.20094 [cs]

    Pith/arXiv arXiv doi:10.48550/arxiv.2406.20094 2025

  59. [60]

    Yue Guo, Tal August, Gondy Leroy, Trevor Cohen, and Lucy Lu Wang. 2024. APPLS: Evaluating Evaluation Metrics for Plain Language Summarization. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (Eds.). Association for Computational Linguistics, Miami, Florida, USA, 91...

    doi:10.18653/v1/2024.emnlp-main.519 2024

  60. [61]

    Simon Hachmeier and Robert Jäschke. 2025. A Benchmark and Robustness Study of In-Context-Learning with Large Language Models in Music Entity Detection. InProceedings of the 31st International Conference on Computational Linguistics. 9845–9859

    2025

  61. [62]

    Anna Hausberger, Hannah Strauss, and Markus Schedl. 2025. ExIM: Exploring Intent of Music Listening for Retrieving User-generated Playlists. In Proceedings of the 2025 ACM SIGIR Conference on Human Information Interaction and Retrieval (CHIIR ’25). Association for Computing Machinery, New York, NY, USA, 348–357. doi:10.1145/3698204.3716470

    arXiv 2025

  62. [63]

    Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua. 2017. Neural collaborative filtering. InProceedings of the 26th international conference on world wide web. 173–182

    2017

  63. [64]

    Zhankui He, Zhouhang Xie, Rahul Jha, Harald Steck, Dawen Liang, Yesu Feng, Bodhisattwa Prasad Majumder, Nathan Kallus, and Julian McAuley

  64. [65]

    Yupeng Hou, Junjie Zhang, Zihan Lin, Hongyu Lu, Ruobing Xie, Julian McAuley, and Wayne Xin Zhao. 2024. Large language models are zero-shot rankers for recommender systems. InEuropean Conference on Information Retrieval. Springer, 364–381

    2024

  65. [66]

    Yifan Hu, Yehuda Koren, and Chris Volinsky. 2008. Collaborative filtering for implicit feedback datasets. In2008 Eighth IEEE international conference on data mining. Ieee, 263–272

    2008

  66. [67]

    Hui Huang, Xingyuan Bu, Hongli Zhou, Yingqi Qu, Jing Liu, Muyun Yang, Bing Xu, and Tiejun Zhao. 2025. An Empirical Study of LLM-as-a-Judge for LLM Evaluation: Fine-tuned Judge Model is not a General Substitute for GPT-4. arXiv:2403.02839 [cs.CL] https://arxiv.org/abs/2403.02839

    Pith/arXiv arXiv 2025

  67. [68]

    Jin Huang, Harrie Oosterhuis, Masoud Mansoury, Herke Van Hoof, and Maarten de Rijke. 2024. Going beyond popularity and positivity bias: Correcting for multifactorial bias in recommender systems. InProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval. 416–426

    2024

  68. [69]

    Wenyu Huang, Guancheng Zhou, Mirella Lapata, Pavlos Vougiouklis, Sebastien Montella, and Jeff Z. Pan. 2025. Prompting large language models with knowledge graphs for question answering involving long-tail facts.Knowledge-Based Systems324 (Aug. 2025), 113648. doi:10.1016/j.knosys. 2025.113648

    arXiv 2025

  69. [70]

    Karim M Ibrahim, Elena V Epure, Geoffroy Peeters, and Gael Richard. 2020. SHOULD WE CONSIDER THE USERS IN CONTEXTUAL MUSIC AUTO-TAGGING MODELS?. In21st International Society for Music Information Retrieval Conference

    2020

  70. [71]

    Ari Jacovi and Yoav Goldberg. 2020. Aligning Faithful Explanations: Towards Explanation as Verification.Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP 2020)(2020), 514–525. https://aclanthology.org/2020.emnlp-main.40

    2020

  71. [72]

    Aryan Jadon and Avinash Patil. 2024. A comprehensive survey of evaluation techniques for recommendation systems. InInternational Conference on Computation of Artificial Intelligence & Machine Learning. Springer, 281–304

    2024

  72. [73]

    Dietmar Jannach, Ahtsham Manzoor, Wanling Cai, and Li Chen. 2021. A survey on conversational recommender systems.ACM Computing Surveys (CSUR)54, 5 (2021), 1–36. Manuscript submitted to ACM Music Recommendation with Large Language Models 31

    2021

  73. [74]

    Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of hallucination in natural language generation.ACM computing surveys55, 12 (2023), 1–38

    2023

  74. [75]

    Marius Kaminskas and Derek Bridge. 2016. Diversity, serendipity, novelty, and coverage: a survey and empirical analysis of beyond-accuracy objectives in recommender systems.ACM Transactions on Interactive Intelligent Systems (TiiS)7, 1 (2016), 1–42

    2016

  75. [76]

    Li Kang, Yuhan Zhao, and Li Chen. 2025. Exploring the Potential of LLMs for Serendipity Evaluation in Recommender Systems. InProceedings of the Nineteenth ACM Conference on Recommender Systems. 746–754

    2025

  76. [77]

    Wang-Cheng Kang and Julian McAuley. 2018. Self-attentive sequential recommendation. In2018 IEEE international conference on data mining (ICDM). IEEE, 197–206

    2018

  77. [78]

    Mohammad Khosravani, Chenyang Huang, and Amine Trabelsi. 2024. Enhancing Argument Summarization: Prioritizing Exhaustiveness in Key Point Generation and Introducing an Automatic Coverage Evaluation Metric. InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volum...

    2024

  78. [79]

    Bart P Knijnenburg and Martijn C Willemsen. 2015. Evaluating recommender systems with user experiments. InRecommender systems handbook. Springer, 309–352

    2015

  79. [80]

    Zhifeng Kong, Arushi Goel, Rohan Badlani, Wei Ping, Rafael Valle, and Bryan Catanzaro. 2024. Audio Flamingo: a novel audio language model with few-shot learning and dialogue abilities. InProceedings of the 41st International Conference on Machine Learning. 25125–25148

    2024

  80. [81]

    Ngoc Luyen Le, Marie-Hélène Abel, and Philippe Gouspillou. 2023. A Constraint-based Recommender System via RDF Knowledge Graphs. In2023 26th International Conference on Computer Supported Cooperative Work in Design (CSCWD). IEEE, Rio de Janeiro, Brazil, 849–854. doi:10.1109/ CSCWD57460.2023.10152701

    arXiv 2023

Showing first 80 references.