From Prompting to Behavioral Alignment: Personalized LLM Judges for Recommendation Evaluation

Alireza S. Ziabari; Colleen Chan; Ding Tong; Kat Ellis

REVIEW 3 major objections 5 minor 17 references

From Prompting to Behavioral Alignment: Personalized LLM Judges for Recommendation Evaluation

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A text-only LLM judge, aligned via supervised fine-tuning and preference optimization on paired rationales, matches a feature-engineered production baseline for recommendation engagement prediction.

desk verdict A useful industry paper with a real empirical finding, but the headline parity claim is asserted without error bars and the ground-truth heuristic is unvalidated. read the letter →

arxiv 2608.11493 v1 pith:LOGSEOTW submitted 2026-08-11 cs.AI cs.LG

Alireza S. Ziabari , Kat Ellis , Colleen Chan , Ding Tong This is my paper · ORCID

classification cs.AIcs.LG

keywords personalizedrecommendationevaluationLLM-as-judgebehavioralalignmentdirectpreferenceoptimizationbidirectionalrationalizationofflinechain-of-thoughtreasoningengagementprediction

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that off-the-shelf LLM judges are unreliable for personalized recommendation evaluation because they can produce factually grounded, internally coherent arguments for both 'play' and 'skip' on the same row and the same user history — a failure mode the authors call bidirectional rationalization. It then claims that prompt engineering alone cannot fix this: the best prompted zero-shot configuration still trails a feature-engineered production baseline. The paper's central proposal is a sequential behavioral alignment recipe: supervised fine-tuning on true reasoning traces followed by direct preference optimization over paired correct and counterfactual rationales. On real homepage interaction logs, this recipe yields a 32.19% Macro-F1 lift over the zero-shot baseline and reaches statistical parity (difference under 0.1%) with the production feature-engineered baseline, while emitting human-readable reasoning traces. A sympathetic reader would take the paper to establish that a text-only LLM evaluator, aligned to observed user behavior, is a viable drop-in replacement for a heavily engineered offline evaluation pipeline on this task.

What carries the argument

The load-bearing mechanism is a two-stage behavioral alignment recipe. First, supervised fine-tuning (SFT) trains the judge on the final label and, in the reasoning variant, on teacher-generated chain-of-thought rationales that match the observed engagement outcome, establishing domain vocabulary and answer format. Second, direct preference optimization (DPO) is applied over paired rationales: for each instance the rationale agreeing with the true action is the chosen response and the rationale arguing the opposite is the rejected response. The pair generation itself exploits bidirectional rationalization — a single capable teacher LLM can write both sides — so preference learning teaches the model which of the many locally valid framings of identical evidence best predicts grounded user behavior.

What would settle it

Take a held-out set of homepage impressions with explicit user-level ground truth, such as eye-tracking or direct feedback, and compare the scroll-heuristic labels to it; if a substantial fraction of 'skip' rows were actually viewed-but-ignored or misattributed, the parity result would need re-evaluation. Alternatively, measure whether the aligned LLM judge's label agreement with true engagement exceeds its agreement with the heuristic labels on a dataset where both are available.

Watch

Extended reading notes

Core claim

The central discovery is a failure mode and its remedy. In zero-shot use, the same LLM, given identical serialized user history, session context, and recommendation row, can argue with equal confidence for play and for skip; after filtering out rationales containing fabricated facts, 77.0% of the paired arguments survive, and 95.4% of the surviving disagreements trace to four recommender-system trade-offs (short- versus long-term signals, calibration versus specialization, popularity versus novelty, exploration versus exploitation). The paper shows that fine-tuning the model on correct rationales plus preference-optimizing it against counterfactual rationales — with reasoning at inference time — collapses this bidirectional rationalizer into a directional judge. On the studied homepage task, the aligned text-based judge matches the feature-engineered production baseline in Macro-F1 (difference less than 0.1%) and improves 32.19% over the zero-shot baseline, with interpretable reasoning traces as a by-product.

Load-bearing premise

The whole comparison rests on the spatial scroll heuristic that labels any row above the played row as a deliberate skip; if that heuristic mislabels exposure as avoidance, the ground-truth labels are noisy and both the production-baseline parity and the DPO preference signal are built on that noise.

Editorial extensions

If this is right

If the parity result holds beyond the study sample, an LLM judge can replace a manually maintained feature pipeline for offline recommendation evaluation, removing feature-engineering overhead.
The 32.19% Macro-F1 lift over the zero-shot baseline shows unaligned LLM judges underperform mostly from decision framing, not missing factual knowledge.
Reasoning-based inference is required to realize the full benefit: SFT+DPO trained with reasoning and inferred with reasoning gives the best result, while the simple inference prompt yields a smaller lift (25.74% versus 32.19%).
The aligned judge's traces expose which user-history signals drive each prediction, adding interpretability that the production feature pipeline does not offer.
The four named trade-offs give a taxonomy of where personalized LLM judges are ambiguous, implying that alignment must resolve these trade-offs rather than merely suppress hallucination.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Editorial inference: because the failure mode is structural (multiple defensible framings of the same evidence), the same paired-rationale preference recipe should transfer to other subjective evaluation tasks with behavioral ground truth, such as ad relevance or content satisfaction, though parity with a feature baseline would need to be re-demonstrated.
Editorial inference: the spatial scroll heuristic's noise level sets an upper bound on agreement between any judge and true user intent; if exposure is mislabeled as a deliberate skip, both the production baseline and the aligned LLM inherit the same noise, so the parity result may partly reflect shared label noise rather than shared understanding.
Editorial inference: a natural extension is to test whether condensing long histories into user profiles — the authors' stated future work — preserves the parity while allowing signals beyond the 50-event cutoff, since raw longer histories showed diminishing returns.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

The paper is worth reading, but not worth taking at face value yet. The genuinely new part is the documentation of 'bidirectional rationalization': for the same user history and same recommended row, a zero-shot LLM can produce factually grounded, internally coherent rationales for both play and skip. They run a factuality filter, and 77% of the pairs survive, meaning the failure is not just hallucination. The four trade-offs they map these rationales onto (short vs long term, calibration vs specialization, novelty vs popularity, exploration vs exploitation) are a plausible and useful taxonomy. That section alone is a decent empirical observation.

The rest is a systematic evaluation of prompting and training strategies. The prompting comparison is thorough, and the finding that simple reasoning beats most structured templates is a useful negative result. The SFT+DPO recipe is standard, but applying it over paired correct/counterfactual rationales is a legitimate adaptation, and the reported 32% Macro-F1 lift over zero-shot is a real number.

The soft spots are where I'd push back. First, the ground-truth labels come from a spatial scroll heuristic: rows above the row the user played are logged as skips. That is exposure-based, not attention-based. Fast scrolling or viewport cropping turns an unseen row into an intentional skip. The paper never validates this against explicit feedback, eye tracking, or anything else. Since the same labels train and evaluate all models, the parity with the production baseline may be parity on a proxy. The paper should either defend this heuristic with data or soften the claim to 'engagement as defined by this heuristic.'

Second, 'statistical parity' is asserted as a difference of <0.1% Macro-F1 with no confidence intervals, no significance test, and no equivalence test. For the central claim, that is not enough. Give the reader the raw baseline score and an error bar.

Third, the factuality filter is described as manual but not documented in detail, and the teacher model is unnamed. This matters because the teacher generates both chosen and rejected rationales; if the teacher has the same framing biases, DPO could be learning those biases. The factuality filter is supposed to guard against that, but it needs more detail.

None of this is fatal. The paper is honest about what it did, and the core result is probably directionally right. But as submitted, the strong claim outruns the evidence.

Who benefits? Researchers and practitioners in recommender-system evaluation, especially anyone thinking about replacing feature pipelines with text-based judges. It deserves a serious referee, but the referee should make the authors pin down the label validation and the statistics.

I'd engage with it. My recommendation: send to peer review with a heavy-revision expectation.

Referee Report

3 major / 5 minor

Summary. This paper studies LLM-based offline judges for personalized recommendation evaluation, formulated as a binary engagement-prediction task on Netflix homepage rows. The authors identify a failure mode they call "bidirectional rationalization": an unaligned LLM can produce factually grounded, internally coherent rationales for both "play" and "skip" from the same evidence, which they trace to four recommender-system trade-offs. To address it, they propose a behavioral alignment recipe: SFT on teacher-generated rationales followed by DPO on paired correct/counterfactual rationales, with reasoning-based inference. On internal Netflix interaction logs, they report a 32.19% Macro-F1 lift over a zero-shot Llama 3.1 8B baseline and a difference of less than 0.1% in Macro-F1 relative to a feature-engineered production baseline. They also report a systematic comparison of prompt formats and training paradigms.

Significance. If the empirical claims hold, the contribution is practically significant: a text-only LLM judge that matches a heavily engineered production evaluator while producing interpretable reasoning traces and avoiding manual feature-pipeline maintenance. The characterization of bidirectional rationalization as structurally distinct from hallucination, supported by a factuality filter, is a useful conceptual contribution. The systematic prompting comparison and the SFT/DPO ablations are also valuable. The paper does not ship code, data, or machine-checked proofs; its strengths are the real production setting, the concrete failure-mode taxonomy, and the clear two-stage alignment recipe. However, the central quantitative claims rest on an unvalidated label heuristic and point estimates without uncertainty quantification, so the significance of the work will depend on closing those gaps.

major comments (3)

[§5.1 and §3] The ground-truth labels used for both training and evaluation are derived from a spatial scroll heuristic (Section 5.1): when a user plays an item from a row, all rows positioned above it are assumed to have been seen and deliberately skipped, and are labeled as negative "skip" events. This heuristic conflates exposure with attention and non-play with rejection; fast scrolling, viewport cropping, and row placement can make an unseen row look like an intentional skip. The same labels are used to construct the Macro-F1 evaluations, the DPO chosen/rejected pairs (Section 4.3.2), and the production baseline comparison, so a biased heuristic directly biases the headline 32.19% lift and the claimed parity. The paper provides no validation of this heuristic against explicit feedback, viewport data, or eye tracking, and no sensitivity or bounded-error analysis. Because every quantitative claim in the paper depends on these labels, this is load-bearing and must be addressed before the parity claim can be accepted.
[§6.2] Section 6.2 states that the best configuration reaches "statistical parity in Macro-F1 score (difference < 0.1%)" with the production baseline. A point estimate of a difference below 0.1% is not statistical parity; the paper reports no confidence intervals, significance test, equivalence test, or per-class breakdown. Macro-F1 on a single test set has sampling variability, and the size and composition of the evaluation set are not reported. The central claim that the aligned LLM judge "matches" the production baseline is therefore asserted rather than demonstrated. Please report bootstrap confidence intervals, an equivalence test with a pre-specified margin, and the number of instances and rows in the evaluation set.
[§3 and §4.4] The claim that bidirectional rationalization is distinct from hallucination rests on a manual filter described only as "manually filtering rationale pairs for unfactual claims." The paper reports that 77.0% (960 of 1,246) of pairs survive and that 95.4% of survivors fit the four trade-offs, but it gives no annotation instructions, no inter-annotator agreement, and no examples of excluded pairs. This matters because the same style of teacher-generated rationale is used as the alignment signal in Section 4.4: if the "correct" rationales are merely plausible post-hoc stories attached to the right label, the SFT and DPO signals are not validated as faithful reasoning. Please document the filtering protocol, report agreement statistics, and provide examples of rejected pairs, or otherwise justify that the rationales reflect genuine decision-relevant evidence.

minor comments (5)

[Table 4] The row for "SFT + DPO Reason" is typeset as "25.7432.190.93 1.05", making the Macro-F1 lift, positive bias, and the two inference-prompt columns indistinguishable; please reformat the table so each cell is clearly separated.
[§3] The sentence "we conducted a qualitative analysis over model's rational for their evaluation" contains a typo ("rational" should be "rationales") and should be rephrased for clarity.
[§5 and §6] The paper does not report the number of instances in the training, validation, and test splits, nor the random seeds or hyperparameters used for SFT and DPO; please include these details to allow replication and to enable readers to assess the stability of the reported lifts.
[§4.4] The teacher model used to generate the paired rationales is described only as a "high-capacity reasoning LLM"; please identify the specific model and version, since the quality of the alignment signal depends on it.
[§6.1] The decision to limit user history to 50 events is reported only in Section 6.1; please state the fixed threshold in the task setup in Section 4.1 so the experimental configuration is described where the data pipeline is defined.

Circularity Check

1 steps flagged · score 3.0 of 10

Partial self-definition in the failure-mode demonstration and the paired-rationale training signal, but the headline parity claim is anchored to an external production baseline and is not circular.

self definitional [Section 3, 'Bidirectional Rationalization'; Section 4.4, 'Generating Reasoning Traces']
"By prompting the model to justify both candidate outcomes, we extracted one confident 'play' rationale and one confident 'skip' rationale per instance, creating a dataset where exactly one reasoning path per pair matches the true user action. ... For each instance, we provided the teacher model with the serialized user history, the recommended row, and a candidate engagement label ('play' or 'skip')."

The 'bidirectional rationalization' failure mode is operationalized by explicitly asking the model to argue for both outcomes, so the existence of two opposing rationales is guaranteed by the elicitation protocol rather than discovered. The paired training data inherit the same construction: the teacher is given a candidate label and asked to rationalize it, and DPO's chosen/rejected distinction is defined solely by which rationale matches the externally supplied engagement label. Thus the motivation for behavioral alignment and the preference signal both reduce to the label-conditioned rationalization procedure itself.

full rationale

The paper's central performance claim — that SFT+DPO with reasoning reaches a 32.19% Macro-F1 lift over zero-shot and statistical parity with a feature-engineered production baseline (difference < 0.1%) — is benchmarked against an external proprietary baseline, so the prediction itself is not derived from its inputs by construction. There are no load-bearing self-citations: the references to prior rationalization and unfaithful-CoT work are external and used as background, and the paper does not invoke a uniqueness theorem from the authors' own prior work. The main circularity-adjacent issue is the self-definitional character of the 'bidirectional rationalization' demonstration: because the protocol asks the model to justify both a 'play' and a 'skip' outcome, the resulting pair of rationales is an artifact of the prompt, and the same label-conditioned teacher rationales are then used as the DPO chosen/rejected signal. That makes the alignment recipe partly feed on the very rationalizer behavior it claims to correct. Separate validity concerns — the unvalidated spatial scroll heuristic used to define ground truth for both training and evaluation, the absence of any stated train/test split, and the assertion of 'statistical parity' without confidence intervals or an equivalence test — are correctness risks rather than circularity, because nothing in the paper reduces the parity comparison to those labels by construction. Overall, the failure-mode motivation is partially self-definitional, but the central external benchmark keeps the paper's main claim non-circular.

Assumptions & free parameters 1 free parameters · 3 assumptions · 1 invented entities

The central empirical results rest on domain assumptions about how scroll behavior maps to engagement labels, and on the use of a teacher LLM's post-hoc rationales as the reasoning ground truth for alignment. The only numerical free parameter visible in the paper is the user-history truncation threshold; many training hyperparameters and evaluation details are unreported. The named construct 'bidirectional rationalization' is an interpretive label for observed behavior rather than an independently validated mechanism.

free parameters (1)

User history truncation threshold = 50 events
Chosen from a length sweep; longer histories up to 50 events improved Macro-F1, then plateaued (Section 6.1). Acts as a model input hyperparameter.

assumptions (3)

domain assumption Rows positioned above a played row were seen and deliberately skipped by the user (spatial scroll heuristic).
Used to construct ground-truth play/skip labels in Sections 3 and 5.1 for both the production baseline and the LLM training signal. If false, labels are systematically noisy.
ad hoc to paper Teacher-generated rationales conditioned on the true engagement label are valid ground truth reasoning paths.
Section 4.4: SFT trains on teacher rationales matching true labels; DPO pairs teacher rationales for play versus skip. No external validation that these rationales reflect the actual user decision process; they are post-hoc justifications produced by a model that is itself a bidirectional rationalizer.
domain assumption The four recommender trade-offs (short versus long term, calibration versus specialization, novelty versus popularity, exploration versus exploitation) explain the surviving bidirectional disagreements.
Section 3 and Table 1: 95.4% of filtered pairs fit one or more of these categories. The taxonomy is interpretive and not formally validated by independent coding or statistical testing.

invented entities (1)

Bidirectional rationalization
purpose: Names and operationalizes the claimed failure mode of unaligned LLM judges, distinct from hallucination.
Defined via an elicitation procedure that prompts for both play and skip rationales, then filters for factual claims. The 77% survival rate measures the procedure's output, not a natural error rate; no external benchmark validates the construct independently of the authors' manual filter.

how reviews work

0 comments

Cite this review

Pith. "Pith review of From Prompting to Behavioral Alignment: Personalized LLM Judges for Recommendation Evaluation." pith.science (2026). https://pith.science/paper/LOGSEOTW

@misc{pith2026260811493,
  author       = {Pith},
  title        = {Pith review of: From Prompting to Behavioral Alignment: Personalized LLM Judges for Recommendation Evaluation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LOGSEOTW}},
  note         = {Machine review of arXiv:2608.11493}
}

read the original abstract

Traditional offline recommendation evaluation relies heavily on complex, manually maintained feature pipelines that are difficult to scale. While Large Language Models (LLMs) offer a promising alternative by predicting user engagement directly from raw text logs, empirical analysis in this study identifies a critical failure mode termed bidirectional rationalization. In a zero-shot setting, LLMs are found to convincingly argue for both positive and negative user engagement outcomes on the exact same item with identical evidence, highlighting the unreliability of off-the-shelf LLMs in predicting user engagement. To resolve this, we develop and apply a sequential behavioral alignment framework pairing fine-tuning with preference optimization over paired correct and counterfactual rationales. Evaluated on real-world homepage interaction logs, this aligned reasoning approach achieves a 32.19\% lift in Macro-F1 score over the zero-shot baseline and matches the production feature-engineered baseline. The results demonstrate that behavioral alignment mitigates bidirectional rationalization while delivering human-interpretable reasoning traces without manual pipeline overhead.

Figures

Figures reproduced from arXiv: 2608.11493 by the authors.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

17 extracted references · 4 canonical work pages

[1]

Bowman, Jan Leike, Jared Kaplan, and Ethan Perez

Yanda Chen, Joe Benton, Ansh Radhakrishnan, Jonathan Uesato, Carson Denison, John Schulman, Arushi Somani, Peter Hase, Misha Wagner, Fabien Roger, Vlad Mikulik, Samuel R. Bowman, Jan Leike, Jared Kaplan, and Ethan Perez. 2025. Reasoning Models Don’t Always Say What They Think. arXiv:2505.05410 [cs.CL]

arXiv 2025
[2]

Francesco Fabbri, Gustavo Penha, Edoardo D’Amico, Alice Wang, Marco De Nadai, Jackie Doremus, Paul Gigioli, Andreas Damianou, Oskar Stål, and Mounia Lalmas. 2025. Evaluating podcast recommendations with profile-aware llm-as-a-judge. InProceedings of the Nineteenth ACM Conference on Recommender Systems. 1181–1186

work page 2025
[3]

Alexandre Gilotte, Clément Calauzènes, Thomas Nedelec, Alexandre Abraham, and Simon Dollé. 2018. Offline a/b testing for recommender systems. InProceed- ings of the eleventh ACM international conference on web search and data mining. 198–206

work page 2018
[4]

Carlos A Gomez-Uribe and Neil Hunt. 2015. The netflix recommender system: Algorithms, business value, and innovation.ACM Transactions on Management Information Systems (TMIS)6, 4 (2015), 1–19

2015
[5]

Zhanyu Liu, Shiyao Wang, Xingmei Wang, Rongzhou Zhang, Jiaxin Deng, Honghui Bao, Jinghao Zhang, et al. 2025. OneRec-Think: In-Text Reasoning for Generative Recommendation. arXiv:2510.11639 [cs.IR]

arXiv 2025
[6]

Pradyumna Shyama Prasad and Minh Nhat Nguyen. 2025. When Two LLMs Debate, Both Think They’ll Win. arXiv:2505.19184 [cs.CL]

arXiv 2025
[7]

Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model.Advances in neural information processing systems36 (2023), 53728–53741

2023
[8]

Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R Bowman, Newton Cheng, Esin Durmus, Zac Hatfield-Dodds, Scott R Johnston, et al. 2023. Towards understanding sycophancy in language models. arXiv preprint arXiv:2310.13548(2023)

arXiv 2023

Show all 17 references

[9]

Harald Steck. 2018. Calibrated Recommendations. InProceedings of the 12th ACM Conference on Recommender Systems (RecSys ’18). 154–162. doi:10.1145/3240323. 3240372

2018 doi
[10]

Miles Turpin, Julian Michael, Ethan Perez, and Samuel R. Bowman. 2023. Lan- guage Models Don’t Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting. InAdvances in Neural Information Processing Systems (NeurIPS)

2023
[11]

Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Lingpeng Kong, Qi Liu, Tianyu Liu, et al. 2024. Large language models are not fair evaluators. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Lo...

2024
[12]

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reason- ing in large language models.Advances in neural information processing systems 35 (2022), 24824–24837

2022
[13]

Dayu Yang, Fumian Chen, and Hui Fang. 2024. Behavior Alignment: A New Perspective of Evaluating LLM-based Conversational Recommender Systems. InProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’24). doi:10.1...

2024
[14]

Qihang Yu, Kairui Fu, Zheqi Lv, Shengyu Zhang, Xinhui Wu, Chen Lin, Feng Wei, Bo Zheng, and Fei Wu. 2026. ThinkRec: Thinking-based Recommendation via LLM. InProceedings of the ACM Web Conference 2026 (WWW ’26)

2026
[15]

Eva Zangerle and Christine Bauer. 2022. Evaluating recommender systems: survey and framework.ACM computing surveys55, 8 (2022), 1–38

2022
[16]

Keyu Zhao, Fengli Xu, and Yong Li. 2025. Reason-to-Recommend: Us- ing Interaction-of-Thought Reasoning to Enhance LLM Recommendation. arXiv:2506.05069 [cs.IR]

2025 arXiv
[17]

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena.Advances in neural information processing systems36 (2023), 46595–46623

2023

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

Bowman, Jan Leike, Jared Kaplan, and Ethan Perez

Yanda Chen, Joe Benton, Ansh Radhakrishnan, Jonathan Uesato, Carson Denison, John Schulman, Arushi Somani, Peter Hase, Misha Wagner, Fabien Roger, Vlad Mikulik, Samuel R. Bowman, Jan Leike, Jared Kaplan, and Ethan Perez. 2025. Reasoning Models Don’t Always Say What They Think. arXiv:2505.05410 [cs.CL]

arXiv 2025

[2] [2]

Francesco Fabbri, Gustavo Penha, Edoardo D’Amico, Alice Wang, Marco De Nadai, Jackie Doremus, Paul Gigioli, Andreas Damianou, Oskar Stål, and Mounia Lalmas. 2025. Evaluating podcast recommendations with profile-aware llm-as-a-judge. InProceedings of the Nineteenth ACM Conference on Recommender Systems. 1181–1186

work page 2025

[3] [3]

Alexandre Gilotte, Clément Calauzènes, Thomas Nedelec, Alexandre Abraham, and Simon Dollé. 2018. Offline a/b testing for recommender systems. InProceed- ings of the eleventh ACM international conference on web search and data mining. 198–206

work page 2018

[4] [4]

Carlos A Gomez-Uribe and Neil Hunt. 2015. The netflix recommender system: Algorithms, business value, and innovation.ACM Transactions on Management Information Systems (TMIS)6, 4 (2015), 1–19

2015

[5] [5]

Zhanyu Liu, Shiyao Wang, Xingmei Wang, Rongzhou Zhang, Jiaxin Deng, Honghui Bao, Jinghao Zhang, et al. 2025. OneRec-Think: In-Text Reasoning for Generative Recommendation. arXiv:2510.11639 [cs.IR]

arXiv 2025

[6] [6]

Pradyumna Shyama Prasad and Minh Nhat Nguyen. 2025. When Two LLMs Debate, Both Think They’ll Win. arXiv:2505.19184 [cs.CL]

arXiv 2025

[7] [7]

Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model.Advances in neural information processing systems36 (2023), 53728–53741

2023

[8] [8]

Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R Bowman, Newton Cheng, Esin Durmus, Zac Hatfield-Dodds, Scott R Johnston, et al. 2023. Towards understanding sycophancy in language models. arXiv preprint arXiv:2310.13548(2023)

arXiv 2023