Pith. sign in

REVIEW 5 major objections 5 minor 115 references

Modeling and Optimizing User Preferences in AI Copilots: A Comprehensive Survey and Taxonomy

T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A survey proposes a unified definition of AI copilots and a three-phase taxonomy of preference optimization.

desk verdict A well-organized survey with a useful pre/mid/post taxonomy, but the post-interaction section rests on placeholder and misattributed references, so the current version cannot be trusted as literature-grounded. read the letter →

arxiv 2505.21907 v2 pith:CYHNEA26 submitted 2025-05-28 cs.AI cs.CLcs.HC

Saleh Afzoon , Zahra Jahanandish , Phuong Thao Huynh , Amin Beheshti , Usman Naseem This is my paper
classification cs.AIcs.CLcs.HC
keywords AIcopilotspreferenceoptimizationpersonalizationuserfeedbacktaxonomyhuman-AIinteractiondirectpersonalizedresponsegeneration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that personalization in AI copilots has a common core: preference optimization, the ability to detect, interpret, and align with what a user wants. It proposes a unified definition of an AI copilot as an interactive, task-oriented system that supports knowledge workers through real-time assistance, domain adaptation, and human-guided decision-making. The central contribution is a taxonomy that organizes preference handling into sources (explicit, implicit, hybrid, interactive, and LLM-generated), detection techniques across pre-, mid-, and post-interaction phases, personalized response generation at prompt, fine-tuning, and architecture levels, and feedback-driven optimization through Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO). If the taxonomy is sound, it gives researchers and builders a shared map of a fragmented design space, turning "make it personalized" into a set of concrete, comparable design decisions.

What carries the argument

The load-bearing object is the paper's conceptual architecture for a preference-aware AI copilot: user input flows into preference sources, then into detection techniques, then into personalized response generation, and finally into feedback-driven optimization, with the loop closing back into the model. The taxonomy splits detection into before (predefined profiles and persona development), during (real-time persona extraction and in-dialogue learning), and after (post-interaction feedback and preference refinement) the interaction. A second named device is the generalized DPO objective, $L(\theta) = -\log\sigma(S(y_w, y_l, x)) + R(y_w, y_l, x)$, which the paper uses to unify DPO and its variants in one table; the scoring function $S$ measures preference between chosen and rejected responses and the regularizer $R$ encodes corrections such as overconfidence penalties or length normalization.

What would settle it

Resolve every reference cited in Sections 4.1-4.4 and compare each attributed claim against the actual source; in particular, check whether [49], [63], [64], and [65] correspond to real published work and whether [64] is distinct from [30]. If the claims tied to those entries cannot be traced, the survey's evidential grounding for the post-interaction feedback category fails.

Watch

Extended reading notes

Core claim

The paper's discovery is that preference optimization can be treated as a system-level lifecycle rather than a single algorithmic step. On its own terms, the paper establishes a conceptual architecture in which user preferences flow from input channels into a preference representation, guide response generation, and then get refined through feedback; within that flow it identifies five sources of preference signals, three temporal phases of detection, three levels of response personalization, and two dominant post-training alignment families (RLHF and DPO). It also distills DPO variants into one generalized objective, $L(\theta) = -\log\sigma(S(y_w, y_l, x)) + R(y_w, y_l, x)$, showing that the many published variants differ only in their scoring function $S$ and a regularization term $R$. The paper's claim is that this unified view is a literature-grounded definition and organizing framework for building user-aligned, persona-aware copilots.

Load-bearing premise

The whole taxonomy stands on the cited literature existing and supporting the specific claims attached to it, and that grounding is visibly fragile in the post-interaction section, where several references carry incomplete identifiers or no verifiable authorship.

Editorial extensions

If this is right

  • Designers can treat preference handling as a lifecycle, choosing sources and techniques per phase rather than bolting a single personalization method onto a chatbot.
  • The taxonomy gives a common vocabulary across recommender systems, human-AI interaction, and LLM alignment, so results from one subfield can be transferred to copilot design.
  • The unified DPO loss form makes variants (CPO, ORPO, SimPO, IRPO, beta-DPO, DPOC, MODPO) directly comparable, helping practitioners select an alignment method by scoring function and regularization.
  • The survey implies that robust personalization needs signals from all three phases, since pre-defined profiles alone are rigid and real-time extraction alone is content-bound.
  • Post-interaction feedback is cast as essential for long-term alignment, pointing to continual learning loops as a core copilot capability.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable next step the authors do not run: use the taxonomy to build an evaluation protocol that measures whether copilots combining pre-, mid-, and post-interaction signals outperform single-phase systems on personalization benchmarks.
  • The taxonomy could be extended to multi-user and team settings, where preferences conflict and must be aggregated or negotiated, a setting the paper only touches via collective behavior modeling.
  • Because several load-bearing references in Section 4.2.3 are unverifiable as printed, the post-interaction feedback category should be regarded as provisional until those sources are confirmed; the taxonomy's other phases rest on standard, traceable literature.
  • One can map each preference source to a measurable cost/benefit trade-off (annotation cost, user fatigue, privacy risk, grounding) and use the paper's tables as a checklist for choosing signals under deployment constraints.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. This manuscript is a survey of preference optimization in AI copilots. It proposes a unified definition of AI copilots, categorizes preference signals into human-driven and LLM-generated sources, organizes detection techniques into pre-, mid-, and post-interaction phases, reviews personalized response generation at prompt, fine-tuning, and architecture levels, and surveys RLHF and DPO alignment methods, including a generalized DPO loss. The main claimed contribution is a literature-grounded taxonomy of preference optimization across the copilot interaction lifecycle.

Significance. If the claims were fully supported, the survey would fill a real gap: it connects user modeling, human-AI interaction, and preference alignment in one framework, and the generalized DPO objective in Section 4.4.2 is a concise unifying device. The comparison tables are helpful for orientation. However, the evidence for the post-interaction stage and several DPO variants rests on citations that are missing, duplicated, or mischaracterized. Because the central claim is literature grounding, these citation problems directly affect the contribution. I credit the authors for a well-organized structure and a useful synthesis of many verifiable sources, but the unresolved references prevent acceptance in the current form.

major comments (5)
  1. [§4.2.3, Table 6] The DPOC (Direct Preference Optimization with Criterion) entry is unsupported. The text states that preference optimization 'using direct feedback criteria collected after interaction' is described in [61] and then names this method DPOC, and Table 6 lists DPOC[115] with the regularization P(r_a,r_b) = −min(0, log r_a − log r_b). However, [61] and [115] are the same paper, 'In dialogues we learn...' (arXiv:2403.03102), which proposes in-dialogue learning (IDL), not a DPO variant with criterion-based penalties. No source in the manuscript supplies the DPOC formula or its name. This removes the evidence for the 'After the Conversation' DPOC row in Table 3 and the DPOC row in Table 6.
  2. [§4.2.2, §4.2.3, Refs [63]–[65]] References [63], [64], and [65] cannot be resolved. Entries [63] and [65] have arXiv identifiers ending in 'XXXX' (2310.XXXX and 2308.XXXX), and [64] has arXiv:2306.XXXX; the existence and content of these works therefore cannot be verified. These references are the sole support for adaptive preference learning strategies in §4.2.2 and for active preference learning and the AFSPP framework in §4.2.3. Without them, the post-interaction refinement discussion is unsubstantiated. The authors must either supply complete, findable references that support each claim or rewrite these paragraphs using verifiable sources.
  3. [Reference [64] vs. [30]] Reference [64] duplicates [30] with a different author list. Both entries are titled 'When to show a suggestion? integrating human feedback in ai-assisted programming'; [30] is the actual AAAI 2024 paper by Mozannar et al., while [64] lists 'Weiyan Xu, Abigail See, et al.' with a placeholder arXiv ID. In §4.2.3, [64] is used to support the claim that suggestion appropriateness is judged from prior user intent and adjustments are made for future turns, which is the topic of [30]. As written, the citation does not refer to a distinct, verifiable source.
  4. [§4.1.1, Ref [49]] The Heimdall citation is not verifiable. The reference is authored only as 'Anonymous' with no arXiv identifier or DOI, so the claim in §4.1.1 that 'frameworks such as Heimdall [49] aggregate and anonymize user data securely' cannot be checked. The authors must provide a full citation with author, venue, and identifier, or the claim should be removed.
  5. [§4.4.2, Table 6, IRPO entry] The IRPO entry does not match the cited paper. The text describes 'Information-Ratio Preference Optimization (IRPO)' with 'auxiliary compression and coverage terms', but reference [113] is 'Iterative Reasoning Preference Optimization' by Pang et al. (NeurIPS 2024), a method for iterative improvement on reasoning tasks. Neither the name nor the regularization described in Table 6 corresponds to [113]. This is a mischaracterization of a listed DPO variant and must be corrected or the entry removed.
minor comments (5)
  1. [§2 heading] The heading 'Priliminary and backgrounds' should read 'Preliminary and Background'.
  2. [References [59], [60], [67]] Several reference entries are incomplete: [59] ('Zhu, Li, Mao, Pandelea, and Cambria'), [60] ('Kong Hao'), and [67] ('Liu Huang, Fu et al.') omit author initials or full names; these should be completed.
  3. [Table 6 caption] The caption says 'Table adapted from [2]', but entries such as DPOC [115] and IRPO [113] do not appear in [2]; the provenance of each row should be clarified.
  4. [§4.4.2, Table 6] The notation r_a and r_b in the DPOC regularization term is used without definition; if the row is retained, these variables should be defined.
  5. [Reference [27]] The phrase 'fouth edition' in reference [27] should be 'fourth edition'.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the survey's taxonomy is literature-assembled, and the broken citations in Section 4.2.3 are evidence-quality failures, not self-referential derivations.

full rationale

This is a survey and taxonomy paper: it makes no quantitative predictions, fits no parameters, and derives no formal results from its own assumptions, so the standard circularity patterns (fitted input called prediction, self-definitional claims, uniqueness imported from authors, ansatz smuggled via citation) do not arise. The central claim is that the AI-copilot definition and the preference-optimization taxonomy are 'literature-grounded'; that grounding is provided by external references such as [2], [5], [29]–[32], and the pre/mid/post-interaction organization is a descriptive framing rather than a derivation from the surveyed methods. The sole DPO equation in Section 4.4.2 is explicitly a generalization that recovers standard DPO and is marked as adapted from the external DPO survey [2], so it is not circular. The only self-citation, PersoBench [68], is paired with the independent survey [69] and is used only to name evaluation aspects (consistency, coherence, diversity); it is not load-bearing for the paper's main contributions. The serious citation-integrity problems in Section 4.2.3 and Table 6—unverifiable references [63]–[65], the duplicated/misattributed [64], and [115] being the IDL paper rather than a DPOC paper—are real evidentiary defects, but they weaken the survey's support base rather than making its argument equivalent to its own inputs. Accordingly, no circular step is identified, and the appropriate score is 0.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The survey relies on domain assumptions about the coherence of the AI copilot category and the completeness of its phase taxonomy. No free parameters or invented entities are introduced because the paper is a conceptual survey. The critical assumption is that the cited references exist and accurately support the claims; multiple entries contain placeholder identifiers or anonymous authorship.

assumptions (3)
  • domain assumption The cited literature exists and supports the claims attributed to it.
    Fails at references [63], [64], [65], which contain placeholder identifiers, and [49], which is anonymous. Section 4.2.3 depends on these for active preference learning and AFSPP.
  • domain assumption AI copilots form a coherent and distinct system class.
    Section 3 defines AI copilot based on heterogeneous prior definitions; the taxonomy assumes this grouping is meaningful and distinct from assistants and chatbots.
  • domain assumption The pre, mid, and post interaction phase decomposition covers preference optimization.
    Section 4.2 partitions preference detection into three phases. This is an editorial choice rather than a result derived from evidence.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Modeling and Optimizing User Preferences in AI Copilots: A Comprehensive Survey and Taxonomy." pith.science (2026). https://pith.science/paper/CYHNEA26

@misc{pith2026250521907,
  author       = {Pith},
  title        = {Pith review of: Modeling and Optimizing User Preferences in AI Copilots: A Comprehensive Survey and Taxonomy},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CYHNEA26}},
  note         = {Machine review of arXiv:2505.21907}
}
read the original abstract

AI copilots represent a new generation of AI-powered systems designed to assist users, particularly knowledge workers and developers, in complex, context-rich tasks. As these systems become more embedded in daily workflows, personalization has emerged as a critical factor for improving usability, effectiveness, and user satisfaction. Central to this personalization is preference optimization: the system's ability to detect, interpret, and align with individual user preferences. While prior work in intelligent assistants and optimization algorithms is extensive, their intersection within AI copilots remains underexplored. This survey addresses that gap by examining how user preferences are operationalized in AI copilots. We investigate how preference signals are sourced, modeled across different interaction stages, and refined through feedback loops. Building on a comprehensive literature review, we define the concept of an AI copilot and introduce a taxonomy of preference optimization techniques across pre-, mid-, and post-interaction phases. Each technique is evaluated in terms of advantages, limitations, and design implications. By consolidating fragmented efforts across AI personalization, human-AI interaction, and language model adaptation, this work offers both a unified conceptual foundation and a practical design perspective for building user-aligned, persona-aware AI copilots that support end-to-end adaptability and deployment.

Figures

Figures reproduced from arXiv: 2505.21907 by the authors.

Figure 1
Figure 1. Conceptual design architecture of a preference-aware AI copilot, illustrating the flow from user input and [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

115 extracted references · 50 canonical work pages

  1. [2]

    A comprehensive survey of direct preference optimization: Datasets, theories, variants, and applications

    Wenyi Xiao, Zechuan Wang, Leilei Gan, Shuai Zhao, Wanggui He, Luu Anh Tuan, Long Chen, Hao Jiang, Zhou Zhao, and Fei Wu. A comprehensive survey of direct preference optimization: Datasets, theories, variants, and applications. arXiv preprint arXiv:2410.15595, 2024

    arXiv

  2. [63]

    Active preference learning for large language models

    Nathan Lee, Arun Suggala, et al. Active preference learning for large language models. arXiv preprint arXiv:2310.XXXX, 2023

    work page 2023

  3. [64]

    When to show a suggestion? integrating human feedback in ai-assisted programming

    Weiyan Xu, Abigail See, et al. When to show a suggestion? integrating human feedback in ai-assisted programming. arXiv preprint arXiv:2306.XXXX, 2023

    work page 2023

  4. [65]

    Afspp: An agent framework for shaping preference and personality with llms

    Lilian Weng et al. Afspp: An agent framework for shaping preference and personality with llms. arXiv preprint arXiv:2308.XXXX, 2023

    work page 2023

  5. [49]

    Heimdall: A privacy-respecting implicit preference collection framework

    Anonymous. Heimdall: A privacy-respecting implicit preference collection framework. In ACM WSDM, 2017

    work page 2017

  6. [115]

    "In Dialogues We Learn": Towards Personalized Dialogue Without Pre-defined Profiles through In-Dialogue Learning

    Chuanqi Cheng, Quan Tu, Shuo Shang, Cunli Mao, Zhengtao Yu, Wei Wu, and Rui Yan. " in dialogues we learn": Towards personalized dialogue without pre-defined profiles through in-dialogue learning. arXiv preprint arXiv:2403.03102, 2024

    work page Pith review arXiv 2024

  7. [30]

    When to show a suggestion? integrating human feedback in ai-assisted programming

    Hussein Mozannar, Gagan Bansal, Adam Fourney, and Eric Horvitz. When to show a suggestion? integrating human feedback in ai-assisted programming. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 10137–10144, 2024

    2024

  8. [113]

    Iterative reasoning preference optimization

    Richard Yuanzhe Pang, Weizhe Yuan, Kyunghyun Cho, He He, Sainbayar Sukhbaatar, and Jason Weston. Iterative reasoning preference optimization. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Advances in Neural Information Processing Systems , volume 37, pages 116617–116637. Curran Associates, Inc., 2024

    work page 2024

Show all 115 references
  1. [1]

    Ai-based digital assistants: Opportunities, threats, and research perspectives

    Alexander Maedche, Christine Legner, Alexander Benlian, Benedikt Berger, Henner Gimpel, Thomas Hess, Oliver Hinz, Stefan Morana, and Matthias Söllner. Ai-based digital assistants: Opportunities, threats, and research perspectives. Business & Information Systems Engineering, 61...

    2019

  2. [3]

    Survey on virtual assistant: Google assistant, siri, cortana, alexa

    Amrita S Tulshan and Sudhir Namdeorao Dhage. Survey on virtual assistant: Google assistant, siri, cortana, alexa. In Advances in Signal Processing and Intelligent Recognition Systems: 4th International Symposium SIRS 2018, Bangalore, India, September 19–22, 2018, Revised Selec...

    2018

  3. [4]

    On the security and privacy challenges of virtual assistants

    Tom Bolton, Tooska Dargahi, Sana Belguith, Mabrook S Al-Rakhami, and Ali Hassan Sodhro. On the security and privacy challenges of virtual assistants. Sensors, 21(7):2312, 2021

    2021

  4. [5]

    Design and evaluation of ai copilots–case studies of retail copilot templates

    Michal Furmakiewicz, Chang Liu, Angus Taylor, and Ilya Venger. Design and evaluation of ai copilots–case studies of retail copilot templates. arXiv preprint arXiv:2407.09512, 2024

    2024 arXiv

  5. [6]

    Computing, cognition and the future of knowing

    John E Kelly. Computing, cognition and the future of knowing. IBM Research. Oct, 13(2015):12, 2015

    2015

  6. [7]

    Foundations of augmented cognition

    Dylan Schmorrow. Foundations of augmented cognition. Springer, 2005

    2005

  7. [8]

    Joint cognitive systems: Foundations of cognitive systems engineering

    Erik Hollnagel and David D Woods. Joint cognitive systems: Foundations of cognitive systems engineering . CRC press, 2005

    2005

  8. [9]

    Experi- mental evidence of effective human–ai collaboration in medical decision-making

    Carlo Reverberi, Tommaso Rigon, Aldo Solari, Cesare Hassan, Paolo Cherubini, and Andrea Cherubini. Experi- mental evidence of effective human–ai collaboration in medical decision-making. Scientific reports, 12(1):14952, 2022

    2022

  9. [10]

    The future of human-ai collaboration: a taxonomy of design knowledge for hybrid intelligence systems

    Dominik Dellermann, Adrian Calma, Nikolaus Lipusch, Thorsten Weber, Sascha Weigel, and Philipp Ebel. The future of human-ai collaboration: a taxonomy of design knowledge for hybrid intelligence systems. arXiv preprint arXiv:2105.03354, 2021

    2021 arXiv

  10. [11]

    Human–ai collaboration enables more empathic conversations in text-based peer-to-peer mental health support

    Ashish Sharma, Inna W Lin, Adam S Miner, David C Atkins, and Tim Althoff. Human–ai collaboration enables more empathic conversations in text-based peer-to-peer mental health support. Nature Machine Intelligence, 5(1):46–57, 2023

    2023

  11. [12]

    Anatomy of a digital assistant

    Stefan Wellsandt, Karl Hribernik, and Klaus-Dieter Thoben. Anatomy of a digital assistant. In Advances in Production Management Systems. Artificial Intelligence for Sustainable and Resilient Production Systems: IFIP WG 5.7 International Conference, APMS 2021, Nantes, France, S...

    2021

  12. [13]

    Classifying smart personal assistants: An empirical cluster analysis

    Robin Knote, Andreas Janson, Matthias Söllner, and Jan Marco Leimeister. Classifying smart personal assistants: An empirical cluster analysis. 2019

    2019

  13. [14]

    what can i help you with?

    Benjamin R Cowan, Nadia Pantidi, David Coyle, Kellie Morrissey, Peter Clarke, Sara Al-Shehri, David Earley, and Natasha Bandeira. " what can i help you with?" infrequent users’ experiences of intelligent personal assistants. In Proceedings of the 19th international conference ...

    2017

  14. [15]

    A literature survey of recent advances in chatbots

    Guendalina Caldarini, Sardar Jaf, and Kenneth McGarry. A literature survey of recent advances in chatbots. Information, 13(1):41, 2022

    2022

  15. [16]

    A survey on privacy issues and solutions for voice-controlled digital assistants

    Luca Hernández Acosta and Delphine Reinhardt. A survey on privacy issues and solutions for voice-controlled digital assistants. Pervasive and Mobile Computing, 80:101523, 2022. 14

    2022

  16. [17]

    Manifestation of virtual assistants and robots into daily life: Vision and challenges

    Reza Rawassizadeh, Taylan Sen, Sunny Jung Kim, Christian Meurisch, Hamidreza Keshavarz, Max Mühlhäuser, and Michael Pazzani. Manifestation of virtual assistants and robots into daily life: Vision and challenges. CCF Transactions on Pervasive Computing and Interaction, 1:163–174, 2019

    2019

  17. [18]

    V oices in and of the machine: Source orientation toward mobile virtual assistants.Computers in Human Behavior, 90:343–350, 2019

    Andrea L Guzman. V oices in and of the machine: Source orientation toward mobile virtual assistants.Computers in Human Behavior, 90:343–350, 2019

    2019

  18. [19]

    Survey on intelligent chatbots: State-of-the-art and future research directions

    Ebtesam H Almansor and Farookh Khadeer Hussain. Survey on intelligent chatbots: State-of-the-art and future research directions. In Complex, Intelligent, and Software Intensive Systems: Proceedings of the 13th International Conference on Complex, Intelligent, and Software Inte...

    2019

  19. [20]

    Review of state-of-the-art design techniques for chatbots

    Ritu Agarwal and Mani Wadhwa. Review of state-of-the-art design techniques for chatbots. SN Computer Science, 1(5):246, 2020

    2020

  20. [21]

    A survey on conversational agents/chatbots classification and design techniques

    Shafquat Hussain, Omid Ameri Sianaki, and Nedal Ababneh. A survey on conversational agents/chatbots classification and design techniques. In Web, Artificial Intelligence and Network Applications: Proceedings of the Workshops of the 33rd International Conference on Advanced Inf...

    2019

  21. [22]

    Chatbots: History, technology, and applications

    Eleni Adamopoulou and Lefteris Moussiades. Chatbots: History, technology, and applications. Machine Learning with Applications, 2:100006, 2020

    2020

  22. [23]

    Improving the domain adaptation of retrieval augmented generation (rag) models for open domain question answering

    Shamane Siriwardhana, Rivindu Weerasekera, Elliott Wen, Tharindu Kaluarachchi, Rajib Rana, and Suranga Nanayakkara. Improving the domain adaptation of retrieval augmented generation (rag) models for open domain question answering. Transactions of the Association for Computatio...

    2023

  23. [24]

    Medical expert systems survey

    Bassem Abu-Nasser. Medical expert systems survey. International Journal of Engineering and Information Systems (IJEAIS), 1(7):218–224, 2017

    2017

  24. [25]

    Expert system methodologies and applications—a decade review from 1995 to 2004

    Shu-Hsien Liao. Expert system methodologies and applications—a decade review from 1995 to 2004. Expert systems with applications, 28(1):93–103, 2005

    1995

  25. [26]

    A survey on expert system in agriculture

    J Joy and K Sreekumar. A survey on expert system in agriculture. International journal of computer science and information technologies, 5:7861–7864, 2014

    2014

  26. [27]

    Expert systems: Principles and programming (fouth edition)

    Joseph Giarratano and Gary Riley. Expert systems: Principles and programming (fouth edition). Canada: Thomson, 2005

    2005

  27. [28]

    A survey of belief rule-base expert system

    Zhi-Jie Zhou, Guan-Yu Hu, Chang-Hua Hu, Cheng-Lin Wen, and Lei-Lei Chang. A survey of belief rule-base expert system. IEEE Transactions on Systems, Man, and Cybernetics: Systems, 51(8):4944–4958, 2019

    2019

  28. [29]

    A multimodal generative ai copilot for human pathology

    Ming Y Lu, Bowen Chen, Drew FK Williamson, Richard J Chen, Melissa Zhao, Aaron K Chow, Kenji Ikemura, Ahrong Kim, Dimitra Pouli, Ankush Patel, et al. A multimodal generative ai copilot for human pathology. Nature, 634(8033):466–473, 2024

    2024

  29. [31]

    Human+ machine: Reimagining work in the age of AI

    Paul R Daugherty and H James Wilson. Human+ machine: Reimagining work in the age of AI. Harvard Business Press, 2018

    2018

  30. [32]

    The rise of the ai co-pilot: Lessons for design from aviation and beyond

    Abigail Sellen and Eric Horvitz. The rise of the ai co-pilot: Lessons for design from aviation and beyond. Communications of the ACM, 67(7):18–23, 2024

    2024

  31. [33]

    Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael I

    Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios N. Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael I. Jordan, Joseph E. Gonzalez, and Ion Stoica. Chatbot arena: An open platform for evaluating llms by human preference. arXiv preprint, 2024

    2024

  32. [34]

    Exploring the potential of generative ai for augmenting choice-based preference elicitation in recommender systems

    Benedikt Loepp and Jürgen Ziegler. Exploring the potential of generative ai for augmenting choice-based preference elicitation in recommender systems. In Adjunct Proceedings of the 32nd ACM Conference on User Modeling, Adaptation and Personalization (UMAP Adjunct ’24), pages 1...

    2024

  33. [35]

    Explicit or implicit feedback? engagement or satisfaction? In Proceedings of the 12th ACM Conference on Recommender Systems (RecSys ’18), pages 24–32, 2018

    Xin Zhao, Meng Wang, Xiangnan He, Ming Gao, and Liqiang He. Explicit or implicit feedback? engagement or satisfaction? In Proceedings of the 12th ACM Conference on Recommender Systems (RecSys ’18), pages 24–32, 2018

    2018

  34. [36]

    Automatic personalization based on web usage mining

    Bamshad Mobasher. Automatic personalization based on web usage mining. ACM SIGWEB Newsletter, 2003

    2003

  35. [37]

    Exploring gaze-based prediction strategies for preference detection in videos

    Melanie Heck, Janick Edinger, Jonathan Bünemann, and Christian Becker. Exploring gaze-based prediction strategies for preference detection in videos. In Proceedings of the 2021 ACM SIGIR Conference on Human Information Interaction and Retrieval (CHIIR), pages 129–138, 2021. 15

    2021

  36. [38]

    Tucker, Kiante Brantley, Adam Cahall, and Thorsten Joachims

    Aaron D. Tucker, Kiante Brantley, Adam Cahall, and Thorsten Joachims. Coactive learning for large language models using implicit user feedback. In Proceedings of the 41st International Conference on Machine Learning (ICML), 2024

    2024

  37. [39]

    Rlhf from heterogeneous feedback via personalization and preference aggregation

    Minsoo Park, Hyunsu Kim, Sungdong Lee, and Seung-won Kim. Rlhf from heterogeneous feedback via personalization and preference aggregation. arXiv preprint, 2024

    2024

  38. [40]

    What are you known for? learning user topical profiles with implicit and explicit footprints

    Xiaowen Cao, Yuan Fang, Feida Zhu, Chuan Zhang, and Ee-Peng Chang. What are you known for? learning user topical profiles with implicit and explicit footprints. In Proceedings of the 10th ACM International Conference on Web Search and Data Mining (WSDM ’17), pages 439–448, 2017

    2017

  39. [41]

    Self-exploring language models: Active preference elicitation for online alignment

    Shenao Zhang, Donghan Yu, Hiteshi Sharma, Han Zhong, Zhihan Liu, Ziyi Yang, Shuohang Wang, Hany Hassan, and Zhaoran Wang. Self-exploring language models: Active preference elicitation for online alignment. In Proceedings of the 41st International Conference on Machine Learning...

    2024

  40. [42]

    Bayesian optimization with llm-based acquisition functions for natural language preference elicitation

    David Eric Austin, Anton Korikov, Armin Toroghi, and Scott Sanner. Bayesian optimization with llm-based acquisition functions for natural language preference elicitation. In Proceedings of the 18th ACM Conference on Recommender Systems (RecSys), 2024

    2024

  41. [43]

    Kunal Handa, Yarin Gal, Ellie Pavlick, Noah Goodman, Jacob Andreas, Alex Tamkin, and Belinda Z. Li. Bayesian preference elicitation with language models. arXiv preprint, 2024

    2024

  42. [44]

    Active preference inference using language models and probabilistic reasoning

    Wasu Top Piriyakulkij, V olodymyr Kuleshov, and Kevin Ellis. Active preference inference using language models and probabilistic reasoning. arXiv preprint, 2023

    2023

  43. [45]

    Evaluating large language models as generative user simulators for conversational recommendation

    Se eun Yoon, Zhankui He, Jessica Maria Echterhoff, and Julian McAuley. Evaluating large language models as generative user simulators for conversational recommendation. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Lin...

    2024

  44. [46]

    Guided profile generation improves personalization with llms

    Yi Zhang, Menghan Liu, Liwei Wang, Qianyu Cheng, and Minlie Huang. Guided profile generation improves personalization with llms. In Findings of the Association for Computational Linguistics: EMNLP 2024, 2024

    2024

  45. [47]

    Aligning language models with preferences through f-divergence minimization

    Kihyuk Go, Jaewook Choe, and Se-Young Kim. Aligning language models with preferences through f-divergence minimization. In Proceedings of the 40th International Conference on Machine Learning (ICML), 2023

    2023

  46. [48]

    Aligning llms with individual preferences via interaction

    Zhen Wu, Fangzhou Li, Yuchen Zhang, Honglak Lee, and Kyunghyun Cho. Aligning llms with individual preferences via interaction. In Proceedings of the 29th International Conference on Computational Linguistics (COLING), 2025

    2025

  47. [50]

    Coached conversational preference elicitation: A case study in understanding movie preferences

    Filip Radlinski, Saravan Chandar, and Hyun Jeon. Coached conversational preference elicitation: A case study in understanding movie preferences. In Proceedings of the 2019 Annual Meeting of the Association for Computational Linguistics (ACL), 2019

    2019

  48. [51]

    Do llms recognize your preferences? evaluating personalized preference following in llms

    Siyan Zhao, Mingyi Hong, Yang Liu, Devamanyu Hazarika, and Kaixiang Lin. Do llms recognize your preferences? evaluating personalized preference following in llms. arXiv preprint, 2025

    2025

  49. [52]

    A survey of user profiling: State-of-the-art, challenges, and solutions

    Christopher Ifeanyi Eke, Azah Anir Norman, Liyana Shuib, and Henry Friday Nweke. A survey of user profiling: State-of-the-art, challenges, and solutions. IEEE Access, 7:144907–144928, 2019

    2019

  50. [53]

    User modeling and user profiling: A comprehensive survey

    Jiahui Zhou, Weinan Zhang, and Jun Wang. User modeling and user profiling: A comprehensive survey. ACM Transactions on Information Systems (TOIS), 40(4):1–49, 2022

    2022

  51. [54]

    Preference learning with gaussian processes

    Wei Chu and Zoubin Ghahramani. Preference learning with gaussian processes. Advances in neural information processing systems, 18, 2005

    2005

  52. [55]

    User persona identification and new service adaptation recommendation

    Jeonghwan Choi, Jaewook Moon, and Jihie Lee. User persona identification and new service adaptation recommendation. In Proceedings of the 29th ACM Conference on User Modeling, Adaptation and Personalization, pages 56–64, 2021

    2021

  53. [56]

    Collaborative filtering to capture ai user’s preferences as norms

    Lakshmi Venkatesh, Shubham Gupta, Molly O’Neill, and Partha Talukdar. Collaborative filtering to capture ai user’s preferences as norms. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, pages 1516–1526, 2022

    2022

  54. [57]

    A survey on accuracy-oriented neural recommendation: From collaborative filtering to information-rich recommendation

    Shuai Zhang, Lina Yao, Aixin Sun, and Yi Tay. A survey on accuracy-oriented neural recommendation: From collaborative filtering to information-rich recommendation. ACM Computing Surveys (CSUR), 54(8):1–38, 2021

    2021

  55. [58]

    Neural collaborative filtering for user preference discovery from biased implicit feedback

    Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua. Neural collaborative filtering for user preference discovery from biased implicit feedback. In WWW, pages 173–182, 2017. 16

    2017

  56. [59]

    Paed- zero-shot persona attribute extraction in dialogues

    Zhu, Li, Mao, Pandelea, and Cambria. Paed- zero-shot persona attribute extraction in dialogues. ACL Anthology, 2023

    2023

  57. [60]

    Enhancing emotional support conversations a framework for dynamic

    Kong Hao. Enhancing emotional support conversations a framework for dynamic. ACL Anthology, 2025

    2025

  58. [62]

    Towards personalized human-ai interaction: Adapting the behavior of ai agents using neural signatures of subjective interest

    Qing Zhao et al. Towards personalized human-ai interaction: Adapting the behavior of ai agents using neural signatures of subjective interest. Proceedings of the National Academy of Sciences, 120(12), 2023

    2023

  59. [66]

    Preferences in ai

    James Lang and Gary Marchant. Preferences in ai. AI & Society, 2020

    2020

  60. [67]

    Learning retrieval augmentation for personalized dialogue generation

    Liu Huang, Fu et al. Learning retrieval augmentation for personalized dialogue generation. arXiv preprint arXiv:2406.18847, 2024

    2024 arXiv

  61. [68]

    Persobench: Benchmarking personalized response generation in large language models

    Saleh Afzoon, Usman Naseem, Amin Beheshti, and Zahra Jamali. Persobench: Benchmarking personalized response generation in large language models. arXiv preprint arXiv:2410.03198, 2024

    2024

  62. [69]

    Recent trends in personalized dialogue generation: A review of datasets, methodologies, and evaluations

    Yi-Pei Chen, Noriki Nishida, Hideki Nakayama, and Yuji Matsumoto. Recent trends in personalized dialogue generation: A review of datasets, methodologies, and evaluations. arXiv preprint arXiv:2405.17974, 2024

    2024 arXiv

  63. [70]

    Cross-graph knowledge exchange for personalized response generation in dialogue systems

    Yuezhou Dong, Ke Qin, Pei Ke, Shuang Liang, and Guangchun Luo. Cross-graph knowledge exchange for personalized response generation in dialogue systems. IEEE Internet of Things Journal, 2025

    2025

  64. [71]

    Context aggregation with topic-focused summarization for personalized medical dialogue generation

    Zhengyuan Liu, Siti Salleh, Pavitra Krishnaswamy, and Nancy Chen. Context aggregation with topic-focused summarization for personalized medical dialogue generation. In Proceedings of the 6th Clinical Natural Language Processing Workshop, pages 310–321, 2024

    2024

  65. [72]

    Will i sound like me? improving persona consistency in dialogues through pragmatic self-consciousness

    Hyunwoo Kim, Byeongchang Kim, and Gunhee Kim. Will i sound like me? improving persona consistency in dialogues through pragmatic self-consciousness. arXiv preprint arXiv:2004.05816, 2020

    2004 arXiv

  66. [73]

    Pk-icr: Persona-knowledge interactive multi-context retrieval for grounded dialogue

    Minsik Oh, Joosung Lee, Jiwei Li, and Guoyin Wang. Pk-icr: Persona-knowledge interactive multi-context retrieval for grounded dialogue. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 16383–16395, 2023

    2023

  67. [74]

    Selective prompting tuning for personalized conversations with llms

    Qiushi Huang, Xubo Liu, Tom Ko, Bo Wu, Wenwu Wang, Yu Zhang, and Lilian Tang. Selective prompting tuning for personalized conversations with llms. arXiv preprint arXiv:2406.18187, 2024

    2024 arXiv

  68. [75]

    Talk to your brain: Artificial personalized intelligence for emotionally adaptive ai interactions

    Sandeep Varma, Shivam Shivam, Sarun Natarajan, Biswarup Ray, Bagesh Kumar, and Om Dabral. Talk to your brain: Artificial personalized intelligence for emotionally adaptive ai interactions. In 2024 IEEE International Conference on Computer Vision and Machine Intelligence (CVMI)...

    2024

  69. [76]

    A cue adaptive decoder for controllable neural response generation

    Weichao Wang, Shi Feng, Wei Gao, Daling Wang, and Yifei Zhang. A cue adaptive decoder for controllable neural response generation. In Proceedings of the Web Conference 2020, pages 2570–2576, 2020

    2020

  70. [77]

    Transfertransfo: A transfer learning approach for neural network based conversational agents

    Thomas Wolf, Victor Sanh, Julien Chaumond, and Clement Delangue. Transfertransfo: A transfer learning approach for neural network based conversational agents. arXiv preprint arXiv:1901.08149, 2019

    1901 arXiv

  71. [78]

    A model-agnostic data manipulation method for persona-based dialogue generation

    Yu Cao, Wei Bi, Meng Fang, Shuming Shi, and Dacheng Tao. A model-agnostic data manipulation method for persona-based dialogue generation. arXiv preprint arXiv:2204.09867, 2022

    2022 arXiv

  72. [79]

    Personapkt: Building personalized dialogue agents via parameter-efficient knowledge transfer

    Xu Han, Bin Guo, Yoon Jung, Benjamin Yao, Yu Zhang, Xiaohu Liu, and Chenlei Guo. Personapkt: Building personalized dialogue agents via parameter-efficient knowledge transfer. arXiv preprint arXiv:2306.08126, 2023

    2023 arXiv

  73. [80]

    Beyond candidates: adaptive dialogue agent utilizing persona and knowledge

    Jungwoo Lim, Myunghoon Kang, Jinsung Kim, Jeongwook Kim, Yuna Hur, and Heui-Seok Lim. Beyond candidates: adaptive dialogue agent utilizing persona and knowledge. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 7950–7963, 2023

    2023

  74. [81]

    Plato: Pre-trained dialogue generation model with discrete latent variable

    Siqi Bao, Huang He, Fan Wang, Hua Wu, and Haifeng Wang. Plato: Pre-trained dialogue generation model with discrete latent variable. arXiv preprint arXiv:1910.07931, 2019

    1910 arXiv

  75. [82]

    Personalized response generation via generative split memory network

    Yuwei Wu, Xuezhe Ma, and Diyi Yang. Personalized response generation via generative split memory network. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1956–1970, 2021. 17

    2021

  76. [83]

    Towards persona-based empathetic conversational models

    Peixiang Zhong, Chen Zhang, Hao Wang, Yong Liu, and Chunyan Miao. Towards persona-based empathetic conversational models. arXiv preprint arXiv:2004.12316, 2020

    2004 arXiv

  77. [84]

    Learning to improve persona consistency in multi-party dialogue generation via text knowledge enhancement

    Dongshi Ju, Shi Feng, Pengcheng Lv, Daling Wang, and Yifei Zhang. Learning to improve persona consistency in multi-party dialogue generation via text knowledge enhancement. In Proceedings of the 29th International Conference on Computational Linguistics, pages 298–309, 2022

    2022

  78. [85]

    Personalized dialogue generation with persona-adaptive attention

    Qiushi Huang, Yu Zhang, Tom Ko, Xubo Liu, Bo Wu, Wenwu Wang, and H Tang. Personalized dialogue generation with persona-adaptive attention. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 12916–12923, 2023

    2023

  79. [86]

    Persona-aware multi-party conversation response generation

    Khyati Mahajan and Samira Shaikh. Persona-aware multi-party conversation response generation. InProceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 12712–12723, 2024

    2024

  80. [87]

    Personalized pieces: Efficient personalized large language models through collaborative efforts

    Zhaoxuan Tan, Zheyuan Liu, and Meng Jiang. Personalized pieces: Efficient personalized large language models through collaborative efforts. arXiv preprint arXiv:2406.10471, 2024

    2024 arXiv

  81. [88]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:2773...

    2022

  82. [89]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36:53728–53741, 2023

    2023

  83. [90]

    Fine-tuning language models from human preferences

    Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593, 2019

    1909 arXiv

  84. [91]

    Learning to summarize with human feedback

    Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea V oss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in neural information processing systems, 33:3008–3021, 2020

    2020

  85. [92]

    Fine-grained human feedback gives better rewards for language model training

    Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A Smith, Mari Ostendorf, and Hannaneh Hajishirzi. Fine-grained human feedback gives better rewards for language model training. Advances in Neural Information Processing Systems, 36:59008–59033, 2023

    2023

  86. [93]

    Fine-tuning language models with reward learning on policy

    Hao Lang, Fei Huang, and Yongbin Li. Fine-tuning language models with reward learning on policy. arXiv preprint arXiv:2403.19279, 2024

    2024 arXiv

  87. [94]

    Pretraining language models with human preferences

    Tomasz Korbak, Kejian Shi, Angelica Chen, Rasika Vinayak Bhalerao, Christopher Buckley, Jason Phang, Samuel R Bowman, and Ethan Perez. Pretraining language models with human preferences. In International Conference on Machine Learning, pages 17506–17533. PMLR, 2023

    2023

  88. [95]

    trlx: A framework for large scale reinforcement learning from human feedback

    Alexander Havrilla, Maksym Zhuravinskyi, Duy Phung, Aman Tiwari, Jonathan Tow, Stella Biderman, Quentin Anthony, and Louis Castricato. trlx: A framework for large scale reinforcement learning from human feedback. In Proceedings of the 2023 Conference on Empirical Methods in Na...

    2023

  89. [96]

    Deep reinforcement learning from human preferences

    Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017

    2017

  90. [97]

    Improving reinforcement learning from human feedback with efficient reward model ensemble

    Shun Zhang, Zhenfang Chen, Sunli Chen, Yikang Shen, Zhiqing Sun, and Chuang Gan. Improving reinforcement learning from human feedback with efficient reward model ensemble. arXiv preprint arXiv:2401.16635, 2024

    2024 arXiv

  91. [98]

    Dense reward for free in reinforcement learning from human feedback

    Alex J Chan, Hao Sun, Samuel Holt, and Mihaela Van Der Schaar. Dense reward for free in reinforcement learning from human feedback. arXiv preprint arXiv:2402.00782, 2024

    2024 arXiv

  92. [99]

    Adaptive preference scaling for reinforcement learning with human feedback

    Ilgee Hong, Zichong Li, Alexander Bukharin, Yixiao Li, Haoming Jiang, Tianbao Yang, and Tuo Zhao. Adaptive preference scaling for reinforcement learning with human feedback. Advances in Neural Information Processing Systems, 37:107249–107269, 2024

    2024

  93. [100]

    Confronting reward model overoptimization with constrained rlhf

    Ted Moskovitz, Aaditya K Singh, DJ Strouse, Tuomas Sandholm, Ruslan Salakhutdinov, Anca D Dragan, and Stephen McAleer. Confronting reward model overoptimization with constrained rlhf. arXiv preprint arXiv:2310.04373, 2023

    2023 arXiv

  94. [101]

    Archer: Training language model agents via hierarchical multi-turn rl

    Yifei Zhou, Andrea Zanette, Jiayi Pan, Sergey Levine, and Aviral Kumar. Archer: Training language model agents via hierarchical multi-turn rl. arXiv preprint arXiv:2402.19446, 2024

    2024 arXiv

  95. [102]

    Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl-constraint

    Wei Xiong, Hanze Dong, Chenlu Ye, Ziqi Wang, Han Zhong, Heng Ji, Nan Jiang, and Tong Zhang. Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl-constraint. arXiv preprint arXiv:2312.11456, 2023. 18

    2023 arXiv

  96. [103]

    Sharp analysis for kl-regularized contextual bandits and rlhf.arXiv preprint arXiv:2411.04625, 2024

    Rui Zhao and et al. Sharp analysis for kl-regularized contextual bandits and rlhf.arXiv preprint arXiv:2411.04625, 2024

    2024 arXiv

  97. [104]

    Statistical rejection sampling improves preference optimization

    Tianqi Liu, Yao Zhao, Rishabh Joshi, Misha Khalman, Mohammad Saleh, Peter J Liu, and Jialu Liu. Statistical rejection sampling improves preference optimization. arXiv preprint arXiv:2309.06657, 2023

    2023 arXiv

  98. [105]

    Rs-dpo: A hybrid rejection sampling and direct preference optimization method for alignment of large language models

    Saeed Khaki, JinJin Li, Lan Ma, Liu Yang, and Prathap Ramachandra. Rs-dpo: A hybrid rejection sampling and direct preference optimization method for alignment of large language models. arXiv preprint arXiv:2402.10038, 2024

    2024 arXiv

  99. [106]

    Self-evolved reward learning for llms

    Chenghua Huang, Zhizhen Fan, Lu Wang, Fangkai Yang, Pu Zhao, Zeqi Lin, Qingwei Lin, Dongmei Zhang, Saravan Rajmohan, and Qi Zhang. Self-evolved reward learning for llms. arXiv preprint arXiv:2411.00418, 2024

    2024 arXiv

  100. [107]

    Ra-pbrl: Provably efficient risk-aware preference-based reinforcement learning

    Yujie Zhao, Jose Aguilar Escamilla, Weyl Lu, and Huazheng Wang. Ra-pbrl: Provably efficient risk-aware preference-based reinforcement learning. Advances in Neural Information Processing Systems, 37:60835–60871, 2024

    2024

  101. [108]

    Exploring intrinsic alignments within text corpus

    Zi Liang, Pinghui Wang, Ruofei Zhang, Haibo Hu, Shuo Zhang, Qingqing Ye, Nuo Xu, Yaxin Xiao, Chen Zhang, and Lizhen Cui. Exploring intrinsic alignments within text corpus. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 27455–27463, 2025

    2025

  102. [109]

    Open problems and fundamental limitations of reinforcement learning from human feedback

    Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, et al. Open problems and fundamental limitations of reinforcement learning from human feedback. arXiv preprint arXiv:2...

    2023 arXiv

  103. [110]

    Contrastive preference optimization: Pushing the boundaries of llm performance in machine translation

    Haoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Kenton Murray, and Young Jin Kim. Contrastive preference optimization: Pushing the boundaries of llm performance in machine translation. arXiv preprint arXiv:2401.08417, 2024

    2024 arXiv

  104. [111]

    Orpo: Monolithic preference optimization without reference model

    Jiwoo Hong, Noah Lee, and James Thorne. Orpo: Monolithic preference optimization without reference model. arXiv preprint arXiv:2403.07691, 2024

    2024 arXiv

  105. [112]

    Simpo: Simple preference optimization with a reference-free reward

    Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with a reference-free reward. Advances in Neural Information Processing Systems, 37:124198–124235, 2024

    2024

  106. [114]

    β-dpo: Direct preference optimization with dynamic β

    Junkang Wu, Yuexiang Xie, Zhengyi Yang, Jiancan Wu, Jinyang Gao, Bolin Ding, Xiang Wang, and Xiangnan He. β-dpo: Direct preference optimization with dynamic β. Advances in Neural Information Processing Systems, 37:129944–129966, 2024

    2024

  107. [116]

    Beyond one-preference-for-all: Multi-objective direct preference optimization

    Zhanhui Zhou, Jie Liu, Chao Yang, Jing Shao, Yu Liu, Xiangyu Yue, Wanli Ouyang, and Yu Qiao. Beyond one-preference-for-all: Multi-objective direct preference optimization. 2023. 19

    2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.