Pith. sign in

REVIEW 3 major objections 6 minor 47 references

LLM-based judges show moderate but reliable alignment with human experts when evaluating conversational music recommendation responses, and clearly outperform string- or embedding-based baselines.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 01:48 UTC pith:QCW663WS

load-bearing objection Honest first-of-its-kind study of LLM judges for CRS response quality, but the bootstrap ignores session-level clustering, so the headline correlations and baseline comparisons are overconfident. the 3 major comments →

arxiv 2607.25640 v1 pith:QCW663WS submitted 2026-07-28 cs.IR

LLM-as-a-Judge for Evaluating System Responses in Conversational Music Recommendation

Seungheon Doh , Bruno Sguerra , Sergio Oramas , Elena V. Epure , Juhan Nam This is my paper
classification cs.IR
keywords LLM-as-a-JudgeConversational Recommendation SystemsResponse EvaluationMusic RecommendationHuman EvaluationPersonalization QualityExplanation QualityCorrelation Analysis
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper asks whether a large language model can stand in for human experts when judging the quality of a conversational music recommender's natural-language responses. Using 20 multi-turn sessions, four response generators, and 400 expert ratings, the authors find that LLM judges correlate with human judgments at Pearson r≈0.55 for personalization and r≈0.51 for explanation—moderate but reliable—while all reference-based metrics (BLEU, ROUGE, BERTScore, embedding similarity) correlate at or below 0.19. The study isolates what makes judging work: access to full conversation history helps most for personalization, and domain-anchored in-context examples help most for explanation. The practical upshot is that LLM-as-a-judge can serve as a cost-effective screening tool, but the human-in-the-loop remains necessary for high-stakes evaluation.

Core claim

The central claim is that LLM-as-a-judge, when conditioned on the user profile, full dialogue history, recommended item, and a rubric with in-context examples, yields moderate positive alignment with domain-expert human ratings for two dimensions of conversational recommendation response quality—Personalization Quality and Explanation Quality—and does so more reliably than any reference-based or reference-free embedding baseline. The best judge reaches bootstrapped Pearson r=0.55 (personalization) and r=0.51 (explanation); lightweight judges still reach r≈0.40–0.43, far above the best baseline r=0.19. The paper also shows that the judge's alignment is driven specifically by conversation hist

What carries the argument

The load-bearing object is the rubric-conditioned judge prompt: a scoring LLM is given the user profile, the multi-turn conversation history, the recommended item, the candidate response, a two-dimension rubric (Personalization Quality, Explanation Quality), and in-context examples, and asked to output scores. Alignment is measured via bootstrapped Pearson and Spearman correlations (10,000 resamples) between judge scores and human expert ratings. A controlled Qwen3-4B comparison isolates generative judging from embedding-based similarity; an ablation on Gemini-3.1Flash-Lite identifies which conditioning components drive alignment.

Load-bearing premise

The human expert ratings—built from only 20 sessions, 80 responses, about five annotations each, with inter-annotator agreement around α=0.45—are treated as a stable ground truth against which the LLM judges are measured.

What would settle it

Re-run the same protocol with, say, 100+ sessions and 10+ annotations per response; if the best LLM judge's correlation with the more reliable human average drops to near zero or below the embedding baselines, the paper's central claim fails. A cheaper falsifier: collect a new set of human ratings on the same 80 responses with independent annotators not using the paper's rubric; if the LLM judges correlate with the original ratings but not with the new ones, the reported alignment is an artifact of the shared rubric.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If LLM judges truly track expert judgment at r≈0.5, they can replace expensive human panels in early-stage or large-scale response screening for conversational recommenders.
  • Reference-based metrics should not be used to compare CRS responses, especially for explanation quality, where they show near-zero correlation.
  • Adding conversation history is the highest-value conditioning step for personalization evaluation; in-context examples are the key lever for explanation evaluation.
  • Lightweight judge models offer a practical cost-accuracy trade-off for routine evaluation.
  • Because stylistic fluency alone can shift judge scores, response generators that differ in writing style cannot be directly compared by LLM judges without controlling for surface form.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If this result transfers to other domains (movies, books, travel), the same conditioning recipe—full dialogue context plus in-context examples—could be the default setup for LLM-as-a-judge in any conversation-grounded generation task.
  • The moderate ceiling (r≈0.55) is partly set by annotator noise (α≈0.45); a study with more annotations per response might show higher true alignment than this point estimate, or reveal that agreement is lower than it looks.
  • The bias case study suggests a practical guardrail: report judge scores alongside a style-variance probe, because fluency inflation can masquerade as quality.
  • One could test whether instructing judges to penalize ornate style, or normalizing for response length, closes the gap between lightweight and full-capacity judges.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper reports a user study evaluating whether LLM-as-a-judge scores correlate with expert human ratings for two dimensions of conversational music recommendation response quality: Personalization and Explanation. Twenty multi-turn sessions from a synthetic dataset are used; four LLMs generate one response each, producing 80 response instances. Twenty music-domain experts provide 400 ratings on a 5-point Likert scale under a rubric with in-context examples that is also given to LLM judges. The paper compares LLM judges against reference-based (BLEU, ROUGE-L, BERTScore, Qwen3-Embedding) and reference-free embedding baselines using 10,000 bootstrap resamples. The main findings are moderate positive Pearson correlations for LLM judges (best r≈0.55 for Personalization and 0.51 for Explanation), outperformance of baselines, and an ablation showing that conversation history and in-context examples are useful conditioning signals. A single-response case study tests semantic inversion, prompt injection, verbosity, and fluency manipulations.

Significance. The contribution is timely and useful. If the results hold, the study provides one of the first empirical estimates of LLM-as-a-judge reliability in the CRS domain, with a carefully controlled comparison of model families/scales and a transparent bootstrap analysis. The use of 20 expert annotators, 80 responses, and two quality dimensions is a reasonable pilot, and the paper explicitly acknowledges the moderate inter-annotator agreement and the resulting ceiling on achievable correlation. The controlled comparison at fixed scale (Qwen3-4B) across reference-based, reference-free, and generative judging is a good design choice. However, because the statistical inference is built on a small number of clusters and the resampling unit is not specified, the headline claims of 'reliable positive correlations' and 'significantly outperform' are not yet established at the level the paper asserts.

major comments (3)
  1. [§3, Table 1] The bootstrap procedure is described only as '10,000 bootstrap simulations' (Section 3, Table 1 caption), with no statement of the resampling unit. The data are nested: 80 responses come from 20 sessions, and responses within a session share dialogue context and overlapping annotators, so they are not independent. If the bootstrap resamples responses independently, the reported 95% CIs understate uncertainty. A session-level (cluster) bootstrap should be reported; based on the reported magnitudes, resampling 20 sessions rather than 80 responses would widen the CI for e.g. Qwen3-LM4B Personalization (r=0.45 [0.17, 0.64]) to approximately [-0.01, 0.80] and would likely make GPT-5.4-nano (r=0.40 [0.16, 0.60]) include zero. This directly affects the claim that 'all LLM-as-a-Judge configurations' show reliable positive correlations. Please report cluster-bootstrap CIs and, ideally, a mixed-ef
  2. [§3, Table 1 and Conclusion] The statement that LLM judges 'significantly outperform' reference-based baselines is not supported by the statistics shown. The 95% CIs for the best baseline and the LLM judges overlap in both dimensions (e.g., Personalization: Qwen3-Embedding r=0.19 [-0.09, 0.45] vs Qwen3-LM4B r=0.45 [0.17, 0.64]; Explanation: Qwen3-Embedding reference-free r=0.30 [0.01, 0.52] vs GPT-5.4 r=0.51 [0.33, 0.66]). Non-overlap of CIs is not required for significance, but the paper reports no paired bootstrap test or other test of the difference between metrics. Please add explicit tests of the difference in correlations (or the difference in scores) with cluster-robust inference.
  3. [§2.4 and §5] The same researcher-authored rubric and in-context examples are provided to both human annotators and LLM judges. This ensures comparability but also means part of the observed agreement may be due to shared instrumentation rather than to the LLM's ability to recover human preferences de novo. The paper states this design choice but does not discuss the threat to external validity. The conclusion that LLM judges are 'a more reliable and cost-effective evaluation strategy' would be strengthened by a validation condition in which humans rate without the supplied rubric (or with a different rubric), or at least by an explicit caveat that the alignment is measured under the specific rubric used here. This is not a circularity in the statistical sense, but it is a scope limitation that should be acknowledged and tested.
minor comments (6)
  1. [Figure 3] The ablation increments are reported as point estimates without significance tests. The 95% CIs likely overlap for adjacent conditioning steps; consider adding pairwise tests or describing the ablation as descriptive only.
  2. [§2.4 / §2.5] The exact judge prompts and in-context examples are not included. For reproducibility, provide them in an appendix or supplementary material.
  3. [Table 1 caption] The caption says 'n=10,000 iterations' but does not state the resampling unit. Please specify whether the bootstrap resamples responses, sessions, or both.
  4. [§2.5] The Qwen3-Embedding reference-free condition includes the user profile, dialogue context, rubric, and examples. This is a strong ablation, but it should be described more explicitly so the reader understands that this baseline is 'reference-free' only in the sense of not using the synthetic reference response.
  5. [§4] The case study uses a single response. This is appropriate for a diagnostic, but the results should not be interpreted as evidence about the distribution of biases across responses or models.
  6. [§2.4] There is a typo: 'Krippendorff's α' appears as 'Krip-pendorff's α'. Also, a brief description of the ordinal Krippendorff's α computation (e.g., distance function) would be helpful.

Circularity Check

0 steps flagged

No significant circularity: empirical alignment measured against independently collected human ratings; minor self-citation is not load-bearing.

full rationale

The paper's central quantities are empirical correlations between LLM-as-a-judge scores and human expert ratings collected in a new user study. The LLM judge receives the user profile, conversation history, recommended item, response, rubric, and in-context examples, and outputs a score; human annotators rate the same responses under the same rubric. No equation maps the inputs to the human score by construction, and no fitted parameter is renamed as a prediction. The only self-citation is the TalkPlayData-Challenge dataset [4] by the same authors, used as a source of conversation sessions; its claimed 'quality comparable to human conversations' is a self-citation but not load-bearing for the alignment result, which rests on this paper's own 400 ratings. The shared rubric between humans and LLMs is a potential source of common-method inflation, but that is a validity threat rather than a definitional equivalence. The bootstrapping with unspecified resampling unit is a statistical robustness concern, outside the circularity categories. The paper honestly reports moderate inter-annotator agreement (alpha~0.45) and low upper-bound correlations (r~0.55), further suggesting no circular inflation.

Axiom & Free-Parameter Ledger

0 free parameters · 4 axioms · 0 invented entities

No fitted numerical parameters; the central quantities are empirical correlations. The study rests on the quality of the human reference, the synthetic data, and the shared rubric. No new physical or mathematical entities are introduced.

axioms (4)
  • domain assumption Human ratings are a valid gold standard despite moderate inter-annotator agreement.
    Section 2.4 reports Krippendorff's α=0.4478/0.4484; the paper treats averaged human scores as the reference y* for correlation. If the ratings are too noisy, all reported correlations are unreliable and the ceiling argument is moot.
  • domain assumption Synthetic TalkPlayData sessions are comparable to real human conversations.
    Section 2.1 relies on prior human evaluation of the synthetic dataset to assert comparable quality; the 20 sampled sessions come from this synthetic corpus, so generalizability to real CRS interactions depends on this premise.
  • ad hoc to paper The researcher-authored rubric and in-context examples define the same evaluation criteria for humans and LLMs.
    Section 2.4: both human annotators and judge LLMs receive prompt P and in-context examples I. This shared instrumentation is a deliberate design choice but can inflate measured alignment because it tests rubric adherence rather than fully independent human preference.
  • standard math Bootstrap resampling of the 80 session-response pairs yields valid confidence intervals.
    Section 3: 10,000 bootstrap iterations are used to estimate Pearson/Spearman correlations and CIs; standard assumption that resampling approximates the sampling distribution, though the effective sample (20 sessions) is small.

pith-pipeline@v1.3.0-alltime-deepseek · 10539 in / 12460 out tokens · 136841 ms · 2026-08-01T01:48:34.632682+00:00 · methodology

0 comments
read the original abstract

Conversational Recommendation Systems (CRS) aim to achieve two primary objectives: recommending relevant items and generating natural language responses. While recommendation accuracy is effectively measured by established ranking metrics, the evaluation of response generation poses a more fundamental challenge. Although human evaluation remains the gold standard, its cost and scalability constraints have motivated the adoption of LLM-as-a-judge as a promising proxy, whose alignment with human judgment in the context of CRS remains an open question. In this paper, we present the first user study to empirically assess the reliability of LLM-as-a-judge for evaluating CRS responses. We sample 20 multi-turn music recommendation sessions and generate candidate system responses using four instruction-tuned LLMs, inducing variance in response quality across model scales. We collect $n{=}400$ ratings from 20 domain-expert annotators, who evaluate each response across two dimensions: Personalization Quality and Explanation Quality. Through bootstrapped correlation analysis, we find that LLM-based judges exhibit moderate positive alignment with human assessments and outperform all reference-based baselines. Furthermore, we analyze how judge performance varies according to model scale and conditioning information, providing practical guidance for deploying LLM-as-a-judge.

Figures

Figures reproduced from arXiv: 2607.25640 by Bruno Sguerra, Elena V. Epure, Juhan Nam, Sergio Oramas, Seungheon Doh.

Figure 1
Figure 1. Figure 1: Proposed LLM-as-a-Judge Pipeline for Conversa [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Frequency distribution of raw human annotation [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Prompt ablation with Gemini-3.1Flash-Lite: condition￾ings added cumulatively (mean and 95% CI). bias for both Personalization and Explanation Quality; Gemini￾3.1Pro also shows a smaller reliable positive bias for Personalization. These results indicate systematic over-scoring in specific judge configurations rather than a uniform tendency across models. We conduct an ablation study using Gemini-3.1Flash-Li… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

47 extracted references · 13 linked inside Pith

  1. [1]

    Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. InProceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization. 65–72

    2005

  2. [2]

    Asli Celikyilmaz, Elizabeth Clark, and Jianfeng Gao. 2020. Evaluation of Text Generation: A Survey.arXiv preprint arXiv:2006.14799(2020)

    Pith/arXiv arXiv 2020

  3. [3]

    Dongping Chen, Ruoxi Chen, Shilin Zhang, Yaochen Wang, Yinuo Liu, Huichi Zhou, Qihui Zhang, Yao Wan, Pan Zhou, and Lichao Sun. 2024. MLLM-as-a- judge: Assessing multimodal llm-as-a-judge with vision-language benchmark. InForty-first International Conference on Machine Learning

    2024

  4. [4]

    Keunwoo Choi, Seungheon Doh, and Juhan Nam. 2025. Talkplaydata 2: An agen- tic synthetic data pipeline for multimodal conversational music recommendation. arXiv preprint arXiv:2509.09685(2025)

    Pith/arXiv arXiv 2025

  5. [5]

    Yea-Seul Choi et al . 2022. To Trust or Not To Trust: How a Conversational Interface Affects Trust, Satisfaction and Persona Assessment in Decision Support Systems. InProceedings of the 2022 CHI Conference on Human Factors in Computing Systems

    2022

  6. [6]

    Pierre Colombo, Guillaume Staerman, Chloé Clavel, and Pablo Piantanida. 2021. Automatic text evaluation through the lens of Wasserstein barycenters. InPro- ceedings of the 2021 Conference on Empirical Methods in Natural Language Pro- cessing. 10450–10466

    2021

  7. [7]

    Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multi- modality, long context, and next generation agentic capabilities.arXiv preprint arXiv:2507.06261(2025)

    Pith/arXiv arXiv 2025

  8. [8]

    SeungHeon Doh, Keunwoo Choi, Daeyong Kwon, Taesu Kim, and Juhan Nam

  9. [9]

    Seungheon Doh, Keunwoo Choi, and Juhan Nam. 2025. Talkplay: Multimodal mu- sic recommendation with large language models.arXiv preprint arXiv:2502.13713 (2025)

    Pith/arXiv arXiv 2025

  10. [10]

    Seungheon Doh, Keunwoo Choi, and Juhan Nam. 2025. Talkplay-tools: Con- versational music recommendation with llm tool calling.arXiv preprint arXiv:2510.01698(2025)

    Pith/arXiv arXiv 2025

  11. [11]

    Seungheon Doh, Junghyun Koo, Marco A Martínez-Ramírez, Woosung Choi, Wei-Hsiang Liao, Qiyu Wu, Juhan Nam, and Yuki Mitsufuji. 2025. LLM2Fx-Tools: Tool Calling For Music Post-Production.arXiv preprint arXiv:2512.01559(2025)

    arXiv 2025

  12. [12]

    1994.An introduction to the bootstrap

    Bradley Efron and Robert J Tibshirani. 1994.An introduction to the bootstrap. Chapman and Hall/CRC

    1994

  13. [13]

    Epure, Yashar Deldjoo, Bruno Sguerra, Markus Schedl, and Manuel Moussallam

    Elena V. Epure, Yashar Deldjoo, Bruno Sguerra, Markus Schedl, and Manuel Moussallam. 2026. Music Recommendation with Large Language Models: Chal- lenges, Opportunities, and Evaluation.ACM Trans. Recomm. Syst.(2026). Just Accepted

    2026

  14. [14]

    Epure and Romain Hennequin

    Elena V. Epure and Romain Hennequin. 2023. A Human Subject Study of Named Entity Recognition in Conversational Music Recommendation Queries. InPro- ceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, Andreas Vlachos and Isabelle Augenstein (Eds.)

    2023

  15. [15]

    2026.Gemini 3.1 Model

    Gemini 3.1 Team, Google. 2026.Gemini 3.1 Model. Technical Report. Google DeepMind. https://deepmind.google/models/gemini/pro/

    2026

  16. [16]

    2026.Gemma 4 Model Card

    Gemma Team, Google. 2026.Gemma 4 Model Card. Technical Report. Google DeepMind. https://ai.google.dev/gemma/docs/core/model_card_4

    2026

  17. [17]

    M Goker and Cynthia Thompson. 2000. The adaptive place advisor: A conversa- tional recommendation system. InProceedings of the 8th German workshop on case based reasoning. 187–198

    2000

  18. [18]

    2026.GPT 5.4 Model

    GPT 5.4, OpenAI. 2026.GPT 5.4 Model. Technical Report. OpenAI. https: //openai.com/index/introducing-gpt-5-4

    2026

  19. [19]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Ab- hishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schel- ten, Alex Vaughan, et al . 2024. The Llama 3 herd of models.arXiv preprint arXiv:2407.21783(2024)

    Pith/arXiv arXiv 2024

  20. [20]

    Takumi Ito, Kees van Deemter, and Jun Suzuki. 2025. Reference-free evaluation metrics for text generation: A survey.arXiv preprint arXiv:2501.12011(2025)

    Pith/arXiv arXiv 2025

  21. [21]

    Gawesh Jawaheer, Peter Weller, and Patty Kostkova. 2014. Modeling user pref- erences in recommender systems: A classification framework for explicit and implicit user feedback.ACM Transactions on Interactive Intelligent Systems (TiiS) 4, 2 (2014), 1–26

    2014

  22. [22]

    Philipp Koehn. 2004. Statistical significance tests for machine translation eval- uation. InProceedings of the 2004 conference on empirical methods in natural language processing. 388–395

    2004

  23. [23]

    2018.Content Analysis: An Introduction to Its Methodology

    Klaus Krippendorff. 2018.Content Analysis: An Introduction to Its Methodology. Sage

    2018

  24. [24]

    Raymond Li, Samira Ebrahimi Kahou, Hannes Schulz, Vincent Michalski, Laurent Charlin, and Chris Pal. 2018. Towards deep conversational recommendations. Advances in neural information processing systems (NeurIPS)(2018)

    2018

  25. [25]

    Chin-Yew Lin. 2004. ROUGE: A Package for Automatic Evaluation of Summaries. InText Summarization Branches Out: Proceedings of the ACL-04 Workshop

    2004

  26. [26]

    Narek Maloyan and Dmitry Namiot. 2025. Adversarial attacks on llm-as-a-judge systems: Insights from prompt injections.arXiv preprint arXiv:2504.18333(2025)

    Pith/arXiv arXiv 2025

  27. [27]

    Jekaterina Novikova, Ondřej Dušek, and Verena Rieser. 2017. Why We Need New Evaluation Metrics for NLG. InProceedings of the 2017 Conference on Em- pirical Methods in Natural Language Processing Workshop on Natural Language Generation

    2017

  28. [28]

    Sergio Oramas, Andres Ferraro, Alvaro Sarasua, and Fabien Gouyon. 2024. Talk- ing to your recs: Multimodal embeddings for recommendation and retrieval. In MuRS 2024: 2nd Music Recommender Systems Workshop

    2024

  29. [29]

    Enrico Palumbo, Gustavo Penha, Andreas Damianou, José Luis Redondo García, Timothy Christopher Heath, Alice Wang, Hugues Bouchard, and Mounia Lal- mas. 2025. Text2Tracks: Prompt-based Music Recommendation via Generative Retrieval.arXiv preprint arXiv:2503.24193(2025)

    Pith/arXiv arXiv 2025

  30. [30]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a Method for Automatic Evaluation of Machine Translation. InProceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL). 311–318

    2002

  31. [31]

    Lasecki, et al

    Anu Rajavenkatanarayanan, Sara Kiesler, Walter S. Lasecki, et al . 2022. UX Research on Conversational Human-AI Interaction: A Literature Review of the ACM Digital Library.Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems(2022)

    2022

  32. [32]

    Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)

    2019

  33. [33]

    MohammadHossein Rezaei and Eduardo Blanco. 2025. Making Language Models Robust Against Negation. InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), Luis Chiruzzo, Alan Ritter, and Lu Wang (Eds.). Association for Computational Linguistics

    2025

  34. [34]

    Markus Schedl, Stefan Brandl, Oleg Lesota, Emilia Parada-Cabaleiro, David Penz, and Navid Rekabsaz. 2022. LFM-2b: A dataset of enriched music listening events for recommender systems research and fairness analysis. InProceedings of the 2022 Conference on Human Information Interaction and Retrieval

    2022

  35. [35]

    Epure, Harin Lee, and Manuel Moussallam

    Bruno Sguerra, Elena V. Epure, Harin Lee, and Manuel Moussallam. 2025. Biases in LLM-generated musical taste profiles for recommendation. InProceedings of the Nineteenth ACM Conference on Recommender Systems. 527–532

    2025

  36. [36]

    Epure, Harin Lee, and Manuel Moussallam

    Bruno Sguerra, Elena V. Epure, Harin Lee, and Manuel Moussallam. 2026. A Study of Biases in LLM-Generated Musical Taste Profiles for Recommendation. ACM Transactions on Recommender Systems(2026)

    2026

  37. [37]

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. 2025. Qwen3 technical report.arXiv preprint arXiv:2505.09388(2025)

    Pith/arXiv arXiv 2025

  38. [38]

    Jiayi Ye, Yanbo Wang, Yue Huang, Dongping Chen, Qihui Zhang, Nuno Moniz, Tian Gao, Werner Geyer, Chao Huang, Pin-Yu Chen, et al . 2024. Justice or prejudice? quantifying biases in llm-as-a-judge.arXiv preprint arXiv:2410.02736 (2024)

    Pith/arXiv arXiv 2024

  39. [39]

    Weinberger, and Yoav Artzi

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi

  40. [40]

    Yongfeng Zhang, Xu Chen, Qingyao Ai, Liu Yang, and W Bruce Croft. 2018. To- wards conversational search and recommendation: System ask, user respond. In Proceedings of the 27th acm international conference on information and knowledge management. 177–186

    2018

  41. [41]

    Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, et al. 2025. Qwen3 embedding: Advancing text embedding and reranking through foundation models.arXiv preprint arXiv:2506.05176(2025)

    Pith/arXiv arXiv 2025

  42. [42]

    Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M Meyer, and Steffen Eger. 2019. MoverScore: Text generation evaluating with contextualized embed- dings and earth mover distance. InProceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-I...

    2019

  43. [43]

    Wei Zhao, Michael Strube, and Steffen Eger. 2023. DiscoScore: Evaluating text generation with BERT and discourse coherence. InProceedings of the 17th Con- ference of the European Chapter of the Association for Computational Linguistics. 3865–3883

    2023

  44. [44]

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judg- ing LLM-as-a-judge with MT-bench and chatbot arena.Advances in neural information processing systems36 (2023), 46595–46623

    2023

  45. [45]

    Hongli Zhou, Hui Huang, Yunfei Long, Bing Xu, Conghui Zhu, Hailong Cao, Muyun Yang, and Tiejun Zhao. 2024. Mitigating the bias of large language model evaluation. InProceedings of the 23rd Chinese National Conference on Computational Linguistics (Volume 1: Main Conference). 1310–1319

    2024

  46. [2020]

    InInternational Con- ference on Learning Representations (ICLR)

    BERTScore: Evaluating Text Generation with BERT. InInternational Con- ference on Learning Representations (ICLR)

  47. [2024]

    Music discovery dialogue generation using human intent analysis and large language models.arXiv preprint arXiv:2411.07439(2024)

    Pith/arXiv arXiv 2024