Pith. sign in

REVIEW 5 major objections 5 minor 1 cited by

PREF claims that a reference-free, two-stage rubric—a general guideline re-ranked by the user's profile—lets an LLM judge score personalised text more accurately than a one-line preference reminder, with +3 to +8 absolute accuracy gains on

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

PREF is a reference-free, two-stage LLM judge that personalizes a quality rubric with a user profile and scores candidates against it, beating reminder-only baselines on the PrefEval implicit preference subset.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection The two-stage rubric is a clean idea, but the headline +4-8 point gain over Reminder may just be a test-split artifact. the 5 major comments →

arxiv 2508.10028 v1 pith:TEDAQYBA submitted 2025-08-08 cs.CL cs.AIcs.HCcs.LG

PREF: Reference-Free Evaluation of Personalised Text Generation in LLMs

Xiao Fu , Hossein A. Rahmani , Bin Wu , Jerome Ramos , Emine Yilmaz , Aldo Lipani This is my paper
classification cs.CL cs.AIcs.HCcs.LG
keywords personalised text generationreference-free evaluationLLM-as-a-judgeevaluation rubricuser preferencesPrefEvalpersonalisation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Personalised text generation matters for user-facing systems, but evaluation is usually one-size-fits-all. The paper proposes PREF, a reference-free evaluation framework that scores candidate answers without gold personalised references. The core move is to split evaluation into a coverage stage—an LLM writes a general, query-specific guideline of quality factors—and a preference stage, where the user's profile re-ranks and augments those factors into a personalised rubric. A scoring LLM then rates the answer against that rubric. On PrefEval's implicit multiple-choice subset, PREF beats a one-line preference reminder by 3–8 absolute points and the zero-shot baseline by more than 55 points across backbone models, with better calibration and ranking quality. The stakes are practical: if PREF works, user-aligned evaluation can be automated, scaled, and made interpretable without collecting gold personalised references.

Core claim

On its own terms, the paper claims that separating what to check from what a given user prioritises is what makes an LLM judge faithful to personalised quality. PREF operationalises this as a three-step pipeline: an LLM generates a general guideline listing universal factors for the query; a preference LLM re-ranks and, when needed, augments these factors into a personalised guideline using the user profile; and a scoring LLM rates the answer against that guideline in the context of the query and profile. The framework never consults a gold reference. Experiments on the PrefEval implicit multiple-choice subset report PREF accuracies of 92–98% across Claude 3 Haiku, GPT-4.1 Mini, LLaMA 3 8B,

What carries the argument

The carrying mechanism is the two-tier rubric. Coverage produces a general guideline: a list of named quality factors (factuality, coherence, completeness, and query-specific criteria) that the scorer should check. Preference then produces a personalised guideline by re-ordering or re-weighting those factors and optionally adding new ones derived from the user profile, so the final rubric encodes both baseline adequacy and user priorities. The scoring LLM evaluates each candidate answer against the personalised guideline, yielding a scalar score. The design keeps coverage reusable across users and preferences reusable across queries, and the resulting rubric doubles as a human-readable expla

Load-bearing premise

The load-bearing premise is that an LLM judge scoring answers against a rubric generated by the same model family reproduces human judgements of personalised quality; the only validation is agreement with PrefEval's gold labels, and there is no fully independent human-rated personalised set.

What would settle it

A direct falsifier would be to run PREF with its preference stage replaced by a random re-ranking of the same general guideline: if random re-ranking matches PREF's accuracy on PrefEval, the user-profile stage is not doing the claimed work. A second check is to use a held-out set of human preference judgments that are not part of PrefEval, including an explicit exclusion (e.g., 'no fish'), and see whether PREF's top-ranked answer agrees with human ratings more than a one-line Reminder prompt does; if it does not, the claim that PREF tracks human judgements fails.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • PREF can be dropped into development pipelines as a reference-free evaluator, enabling repeated regression testing of personalised generation as prompts or user cohorts change.
  • Smaller open-weights models paired with PREF approximate the evaluation quality of much larger models, shrinking serving cost for user-aligned evaluation.
  • The personalised guideline is auditable: a developer or user can see which factors drove a low score and adjust either the answer or the preference weights.
  • Separating coverage from preference improves score calibration and ranking quality over a single coverage-only rubric, especially for smaller backbones.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • [Editorial inference] The paper validates PREF only on PrefEval's implicit multiple-choice subset of open-domain QA; whether the two-stage rubric transfers to long-form generation, code review, or multimodal output is untested and would require richer coverage factors.
  • [Editorial inference] Because both the rubric and the score come from the same LLM family, a stronger test would use one model to build the guideline and a different model to score; the paper's design assigns one backbone to all three roles, so cross-model transfer is unknown.
  • [Editorial inference] The positive correlation between preference-conditioned factor rankings and human explanations suggests the factor rankings themselves could serve as synthetic preference labels for training evaluators, but the paper does not explore this.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper introduces PREF, a reference-free evaluation framework for personalised text generation. It operates in three stages: (1) a coverage LLM generates a query-specific general guideline listing quality factors; (2) a preference LLM re-ranks and augments these factors using a user profile to produce a personalised rubric; (3) a scoring LLM rates candidate answers against this rubric. The authors evaluate on the PrefEval implicit multiple-choice subset, comparing PREF accuracy to Zero-shot and Reminder baselines across three or four backbones, and report MSE, nDCG, an ablation of the user-preference stage, and an explainability analysis correlating factor rankings with human explanations. They claim consistent accuracy gains (+3–8 points over Reminder, +55 over Zero-shot), better calibration, and that smaller models can approach larger ones.

Significance. Assuming the empirical claims hold, the two-tier rubric idea is a useful contribution: it separates generic quality from user-specific priority, produces an auditable rubric, and operates without gold references. The paper also provides an ablation and a factor-ranking analysis that goes beyond simple accuracy. However, the current evidence is not yet convincing: the headline comparison to Reminder/Zero-shot is uncontrolled, no significance tests are reported, and nDCG is near ceiling. The method itself is not circular in a harmful sense, since the gold labels are external to the rubric generation. With a properly controlled evaluation and additional baselines, this could be a solid contribution.

major comments (5)
  1. [Data and Table 1] The central claim that PREF outperforms Reminder by +3–8 points is not supported by the experimental design. The Data section states that the authors hold out 20% (200 questions) as a test set, while Table 1's caption says Zero-shot and Reminder results are 'taken from the PrefEval paper'. Unless those published numbers were computed on exactly the same 200-question split, with the same backbone versions, prompts, and temperature=0 setting, the comparison is uncontrolled and the observed gap could be due to test-set difficulty or prompt differences rather than to PREF. The authors must re-run the baselines on their own split (or use the PrefEval official split) and report the comparison under identical conditions.
  2. [Table 1 and Table 2] No confidence intervals or significance tests are reported anywhere in the paper. With only 200 test questions, the +3–8 point accuracy differences in Table 1 may well be within sampling noise. The authors should report paired significance tests (e.g., McNemar's test) or bootstrap confidence intervals for the accuracy differences. The nDCG values in Table 2 are also near ceiling (0.9742–0.9996), which limits their discriminative value; the MSE metric is based on an arbitrary 10/0 mapping of gold/distractors and may not be comparable across models with different output scales.
  3. [Table 1 vs. Table 2] The paper claims to evaluate PREF with four backbones (Claude 3 Haiku, GPT-4.1 Mini, LLaMA 3 8B, LLaMA 3 70B), but Table 1 lists only three and omits GPT-4.1 Mini. The text states that 'across all backbones' PREF outperforms Reminder by +3–8 points, yet no Reminder or Zero-shot numbers are shown for GPT-4.1 Mini. The authors should either add GPT-4.1 Mini to Table 1 or explicitly qualify the claim to the three backbones for which the comparison exists.
  4. [Algorithm 1 and Experimental Setup] The method is under-specified for replication. Algorithm 1 gives a high-level loop, but the actual prompts used for the coverage, preference, and scoring stages are not included in the paper or an appendix. Without these prompts, readers cannot reproduce the results or judge whether the guidelines are truly 'comprehensive' or the re-ranking is meaningful. Since code is not released at submission time, the authors should provide the exact prompt templates (including any few-shot examples and output parsing instructions) in the supplementary material.
  5. [Related Work and Experiments] The paper positions PREF against prior LLM-as-judge evaluators (AuPEL, PerSE) but does not include them as baselines in the experiments. Given that these methods are also reference-free and personalised, omitting them undermines the claim of state-of-the-art performance. The authors should either compare against these methods on the same split or justify their exclusion (e.g., lack of public implementation or incompatibility with PrefEval's format).
minor comments (5)
  1. [Explainability, Table 3] The correlations in Table 3 are described as 'statistically substantial' but no p-values or confidence intervals are provided. Please report them.
  2. [Beyond accuracy: MSE and nDCG] The 10/0 mapping for gold/distractors is arbitrary. As the LLM judge may use a different internal scale, MSE values may not be comparable across backbones. Please justify or use rank-based calibration.
  3. [Abstract vs. Section 'From Personalisation to Evaluation'] The abstract says 'three-step pipeline' while the body calls it a 'two-stage evaluation framework' plus scoring; please align terminology.
  4. [Throughout] Typo: 'na¨ıve' should be 'naïve' (or 'naive'). Also, 'G ROUP LENS' in Related Work contains stray spaces; fix formatting.
  5. [Figure 3] The ablation bar chart is not included in the text; ensure the figure is legible and the text reports the exact numbers for all conditions.

Circularity Check

0 steps flagged

No significant circularity: PREF's accuracy is validated against PrefEval's external human gold labels, and no fitted parameter is renamed as a prediction.

full rationale

PREF's central claim is that its reference-free, two-stage rubric scores align with human personalized judgments. This is tested against the PrefEval benchmark, whose gold labels are human-annotated and external to PREF. The rubric is constructed from the query and user preference before any candidate answer is scored; the gold answer is never used to generate, re-rank, or calibrate the rubric. It is used only after scoring to compute accuracy, MSE, and nDCG. No parameter is fitted to the human labels and then reported as a prediction. The fact that the same backbone model generates the rubric, personalizes it, and applies it is a self-referential design choice that may introduce bias, but it does not make the result true by construction—the target remains PrefEval's independently collected human labels. The comparison to Reminder and Zero-shot relies on numbers imported from the PrefEval paper, possibly computed on a different 20% split; this is a serious experimental-control concern, but it is not circularity because PREF's own numbers are not derived from those baselines. Self-citations in Related Work are not load-bearing for the empirical claim. No specific circular step can be quoted with a reduction to the paper's own inputs, so the appropriate circularity score is 0.

Axiom & Free-Parameter Ledger

0 free parameters · 4 axioms · 0 invented entities

No free parameters are fitted. The framework rests on domain assumptions about user attention, LLM rubric coverage, faithful LLM judging, and the validity of the benchmark as a proxy for human personalized quality.

axioms (4)
  • domain assumption Assumption 2 (Bounded attention): users prefer answers that foreground information aligned with their personal preferences.
    Design principle in the section 'From Personalisation to Evaluation', motivates the personalized guideline stage.
  • domain assumption The coverage LLM can enumerate a sufficient set of quality factors for a query so that the preference stage can correct omissions via augmentation.
    Central to Stage 1; if coverage is poor, augmentation in Stage 2 may not fully compensate.
  • domain assumption The scoring LLM faithfully follows the rubric and its guardrails, including not excusing factual errors.
    The scoring stage is an LLM judge; the paper acknowledges LLM bias and hallucination in Limitations.
  • domain assumption PrefEval implicit multiple-choice gold labels are a valid proxy for human personalized quality.
    All accuracy and calibration claims are computed against this benchmark in the Data section.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of PREF: Reference-Free Evaluation of Personalised Text Generation in LLMs." pith.science (2026). https://pith.science/paper/TEDAQYBA

@misc{pith2026250810028,
  author       = {Pith},
  title        = {Pith review of: PREF: Reference-Free Evaluation of Personalised Text Generation in LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TEDAQYBA}},
  note         = {Machine review of arXiv:2508.10028}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Personalised text generation is essential for user-centric information systems, yet most evaluation methods overlook the individuality of users. We introduce \textbf{PREF}, a \textbf{P}ersonalised \textbf{R}eference-free \textbf{E}valuation \textbf{F}ramework that jointly measures general output quality and user-specific alignment without requiring gold personalised references. PREF operates in a three-step pipeline: (1) a coverage stage uses a large language model (LLM) to generate a comprehensive, query-specific guideline covering universal criteria such as factuality, coherence, and completeness; (2) a preference stage re-ranks and selectively augments these factors using the target user's profile, stated or inferred preferences, and context, producing a personalised evaluation rubric; and (3) a scoring stage applies an LLM judge to rate candidate answers against this rubric, ensuring baseline adequacy while capturing subjective priorities. This separation of coverage from preference improves robustness, transparency, and reusability, and allows smaller models to approximate the personalised quality of larger ones. Experiments on the PrefEval benchmark, including implicit preference-following tasks, show that PREF achieves higher accuracy, better calibration, and closer alignment with human judgments than strong baselines. By enabling scalable, interpretable, and user-aligned evaluation, PREF lays the groundwork for more reliable assessment and development of personalised language generation systems.

Figures

Figures reproduced from arXiv: 2508.10028 by Aldo Lipani, Bin Wu, Emine Yilmaz, Hossein A. Rahmani, Jerome Ramos, Xiao Fu.

Figure 1
Figure 1. Figure 1: PREF at a glance. Venn diagrams illustrating (left) the relationship between an ideal, exhaustive answer and a per￾sonalised answer to the same query; (centre) the relationship between an ideal, general guideline and a user-specific selection (or ordering) of factors; and (right) the practical regime adopted by PREF, where the ranker may add extra factors to craft a personalised guideline when the general … view at source ↗
Figure 2
Figure 2. Figure 2: The PREF scoring pipeline. A three-step process—coverage, preference (User profiles), and scoring—maps a query q, user profile p, and candidate answer a to a final personalised score sa with a real example on PrefEval. Model Accuracy ↑ MSE ↓ nDCG ↑ Claude 3 Haiku 0.92 5.27 0.9899 GPT-4.1 Mini 0.98 2.19 0.9996 LLaMA 3 8B 0.92 6.62 0.9742 LLaMA 3 70B 0.97 1.87 0.9980 [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Ablation on the user-preference (UP). w/ UP: full PREF pipeline; w/o UP: answers are scored only against the general guideline from Stage one. Bars report accuracy, MSE (lower is better), and nDCG (higher is better) on the PrefEval implicit multiple-choice split. for LLaMA 3 8B and GPT-4.1 Mini. For other models (Claude 3 Haiku, LLaMA 3 70B) the change is marginal, suggesting that they already approximate … view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Preference-Aware Rubric Learning for Personalized Evaluation

    cs.CL 2026-05 unverdicted novelty 6.0

    PARL formulates personalized LLM evaluation as a learning problem that induces preference-aware rubrics from raw user histories via discriminative RL and self-validation.

Reference graph

Works this paper leans on

29 extracted references · 13 canonical work pages · cited by 1 Pith paper

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    L.; Almeida, D.; Altenschmidt, J.; Altman, S.; Anadkat, S.; et al

    Achiam, J.; Adler, S.; Agarwal, S.; Ahmad, L.; Akkaya, I.; Aleman, F. L.; Almeida, D.; Altenschmidt, J.; Altman, S.; Anadkat, S.; et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

    Pith/arXiv arXiv 2023

  4. [4]

    Afzoon, S.; Naseem, U.; Beheshti, A.; and Jamali, Z. 2024. PersoBench: Benchmarking Personalized Response Generation in Large Language Models. CoRR, abs/2410.03198

    arXiv 2024

  5. [5]

    Brown, T.; Mann, B.; Ryder, N.; Subbiah, M.; Kaplan, J. D.; Dhariwal, P.; Neelakantan, A.; Shyam, P.; Sastry, G.; Askell, A.; Agarwal, S.; Herbert-Voss, A.; Krueger, G.; Henighan, T.; Child, R.; Ramesh, A.; Ziegler, D.; Wu, J.; Winter, C.; Hesse, C.; Chen, M.; Sigler, E.; Litwin, M.; Gray, S.; Chess, B.; Clark, J.; Berner, C.; McCandlish, S.; Radford, A.;...

    work page 2020

  6. [6]

    Brusilovsky, P. 2001. Adaptive hypermedia. User modeling and user-adapted interaction, 11(1): 87--110

    work page 2001

  7. [7]

    Dathathri, S.; Madotto, A.; Lan, J.; Hung, J.; Frank, E.; Molino, P.; Yosinski, J.; and Liu, R. 2020. Plug and Play Language Models: A Simple Approach to Controlled Text Generation. In International Conference on Learning Representations (ICLR 2020). ArXiv:1912.02164

    Pith/arXiv arXiv 2020

  8. [8]

    H.; Taylor, C

    Jiang, B.; Hao, Z.; Cho, Y.; Li, B.; Yuan, Y.; Chen, S.; Ungar, L. H.; Taylor, C. J.; and Roth, D. 2025. Know Me, Respond to Me: Benchmarking LLMs for Dynamic User Profiling and Personalized Responses at Scale. CoRR, abs/2504.14225

    arXiv 2025

  9. [9]

    R.; Vidgen, B.; R \"o ttger, P.; and Hale, S

    Kirk, H. R.; Vidgen, B.; R \"o ttger, P.; and Hale, S. A. 2024. The benefits, risks and bounds of personalizing the alignment of large language models to individuals. Nature Machine Intelligence, 6(4): 383--392

    work page 2024

  10. [10]

    A.; Dernoncourt, F.; Deilamsalehy, H.; Chen, X.; Zhang, R.; Agarwal, S.; Lipka, N.; Nguyen, C

    Kumar, I.; Viswanathan, S.; Yerra, S.; Salemi, A.; Rossi, R. A.; Dernoncourt, F.; Deilamsalehy, H.; Chen, X.; Zhang, R.; Agarwal, S.; Lipka, N.; Nguyen, C. V.; Nguyen, T. H.; and Zamani, H. 2024. LongLaMP: A Benchmark for Personalized Long-form Text Generation. arXiv preprint arXiv:2407.11016

    Pith/arXiv arXiv 2024

  11. [11]

    Li, J.; Galley, M.; Brockett, C.; Spithourakis, G.; Gao, J.; and Dolan, B. 2016. A Persona-Based Neural Conversation Model. In Erk, K.; and Smith, N. A., eds., Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 994--1003. Berlin, Germany: Association for Computational Linguistics

    work page 2016

  12. [12]

    Li, X.; Zhang, T.; Dubois, Y.; Taori, R.; Gulrajani, I.; Guestrin, C.; Liang, P.; and Hashimoto, T. B. 2023. AlpacaEval: An Automatic Evaluator of Instruction-following Models. https://github.com/tatsu-lab/alpaca_eval

    work page 2023

  13. [13]

    C.; and Leqi, L

    Li, X.; Zhou, R.; Lipton, Z. C.; and Leqi, L. 2024. Personalized language modeling from personalized human feedback. arXiv preprint arXiv:2402.05133

    Pith/arXiv arXiv 2024

  14. [14]

    L.; and Liang, P

    Li, X. L.; and Liang, P. 2021. Prefix-Tuning: Optimizing Continuous Prompts for Generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics (ACL–IJCNLP 2021), 4582--4597

    work page 2021

  15. [15]

    Lin, C.-Y. 2004. ROUGE : A Package for Automatic Evaluation of Summaries. In Text Summarization Branches Out, 74--81. Barcelona, Spain: Association for Computational Linguistics

    2004

  16. [16]

    Papineni, K.; Roukos, S.; Ward, T.; and Zhu, W.-J. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, 311--318

    2002

  17. [17]

    A.; Wang, X.; Fu, X.; and Lipani, A

    Ramos, J.; Rahmani, H. A.; Wang, X.; Fu, X.; and Lipani, A. 2024. Transparent and Scrutable Recommendations Using Natural Language User Profiles. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024), 13971--13984

    work page 2024

  18. [18]

    Ramos, J.; Wu, B.; and Lipani, A. 2024. PeaPOD: Personalized Prompt Distillation for Generative Recommendation. arXiv preprint arXiv:2407.05033

    work page arXiv 2024

  19. [19]

    Resnick, P.; Iacovou, N.; Suchak, M.; Bergstrom, P.; and Riedl, J. 1994. Grouplens: An open architecture for collaborative filtering of netnews. In Proceedings of the 1994 ACM conference on Computer supported cooperative work, 175--186

    work page 1994

  20. [20]

    Salemi, A.; Mysore, S.; Bendersky, M.; and Zamani, H. 2023. LaMP: When Large Language Models Meet Personalization. arXiv preprint arXiv:2304.11406

    Pith/arXiv arXiv 2023

  21. [21]

    Tan, Z.; Zeng, Q.; Tian, Y.; Liu, Z.; Yin, B.; and Jiang, M. 2024. Democratizing Large Language Models via Personalized Parameter-Efficient Fine-tuning. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP 2024). ArXiv:2402.04401

    Pith/arXiv arXiv 2024

  22. [22]

    Van Der Lee, C.; Gatt, A.; Van Miltenburg, E.; Wubben, S.; and Krahmer, E. 2019. Best practices for the human evaluation of automatically generated text. In Proceedings of the 12th international conference on natural language generation, 355--368

    work page 2019

  23. [23]

    Wang, D.; Yang, K.; Zhu, H.; Yang, X.; Cohen, A.; Li, L.; and Tian, Y. 2024. Learning Personalized Alignment for Evaluating Open-ended Text Generation. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP 2024), 13274--13292. Miami, USA

    work page 2024

  24. [24]

    Wang, Y.; Jiang, J.; Zhang, M.; Li, C.; Liang, Y.; Mei, Q.; and Bendersky, M. 2023. Automated Evaluation of Personalized Text Generation using Large Language Models. arXiv preprint arXiv:2310.11593

    Pith/arXiv arXiv 2023

  25. [25]

    Q.; and Artzi, Y

    Zhang, T.; Kishore, V.; Wu, F.; Weinberger, K. Q.; and Artzi, Y. 2020. BERTScore: Evaluating Text Generation with BERT . In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020 . OpenReview.net

    2020

  26. [26]

    A.; Kveton, B.; Shao, Y.; Yang, D.; Zamani, H.; Dernoncourt, F.; Barrow, J.; Yu, T.; Kim, S.; et al

    Zhang, Z.; Rossi, R. A.; Kveton, B.; Shao, Y.; Yang, D.; Zamani, H.; Dernoncourt, F.; Barrow, J.; Yu, T.; Kim, S.; et al. 2024. Personalization of large language models: A survey. arXiv preprint arXiv:2411.00027

    Pith/arXiv arXiv 2024

  27. [27]

    Zhao, S.; Hong, M.; Liu, Y.; Hazarika, D.; and Lin, K. 2025 a . Do LLMs Recognize Your Preferences? Evaluating Personalized Preference Following in LLMs. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025 . OpenReview.net

    work page 2025

  28. [28]

    B.; and Yilmaz, E

    Zhao, Z.; Vania, C.; Kayal, S.; Khan, N.; Cohen, S. B.; and Yilmaz, E. 2025 b . PersonaLens: A Benchmark for Personalization Evaluation in Conversational AI Assistants. arXiv preprint arXiv:2506.09902

    Pith/arXiv arXiv 2025

  29. [29]

    P.; Zhang, H.; Gonzalez, J

    Zheng, L.; Chiang, W.; Sheng, Y.; Zhuang, S.; Wu, Z.; Zhuang, Y.; Lin, Z.; Li, Z.; Li, D.; Xing, E. P.; Zhang, H.; Gonzalez, J. E.; and Stoica, I. 2023. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. In Oh, A.; Naumann, T.; Globerson, A.; Saenko, K.; Hardt, M.; and Levine, S., eds., Advances in Neural Information Processing Systems 36: Annual Con...

    work page 2023

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.