Pith. sign in

REVIEW 3 major objections 5 minor 2 cited by

The paper claims that on-demand synthetic multi-turn interaction data can replace scarce real personalization logs, and that a prompt-rewriting optimizer trained on that data can personalize a fixed LLM with no parameter access, producing a

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 23:48 UTC pith:YDFW6APF

load-bearing objection PPOpt is a well-executed systems paper whose headline personalization gains are measured against the same synthetic persona-judge coupling used in training — the core claim needs external validation. the 3 major comments →

arxiv 2602.12394 v2 pith:YDFW6APF submitted 2026-02-12 cs.LG

Synthetic Interaction Data for Scalable Personalization in Large Language Models

Yuchen Ma , Yue Huang , Wenjie Wang , Xiaonan Luo , Xiangliang Zhang , Stefan Feuerriegel This is my paper
classification cs.LG
keywords personalized promptingsynthetic interaction dataprompt optimizationmulti-turn dialoguereinforcement learningLLM personalizationuser profile inferencenoise-robust training
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The authors argue that the bottleneck to personalized LLM use is data, not model capacity. Because real personalized interaction logs are scarce and privacy-sensitive, they build a simulator (PersonaGym) that generates realistic multi-turn conversations between a persona-driven User, an Assistant, and a Distractor that injects human-like noise. From this synthetic data (PersonaAtlas) they train PPOpt, a prompt optimizer that first infers a user profile from interaction history, then rewrites the user's current query to match that profile, without touching the deployed model's weights. The central claim is that personalization can be moved entirely into prompt rewriting and that the training signal can be scaled by generating more synthetic dialogues. A sympathetic reader would care because this reframes personalization from a per-user fine-tuning problem into a data-generation plus prompt-optimization problem that works even for closed, proprietary models.

Core claim

The paper's central discovery is that a prompt-level optimizer trained on synthetic multi-turn interaction data can deliver large, consistent personalization gains across many deployed LLMs while preserving task performance, all without access to model parameters or gradients. PersonaGym treats each user as a dynamic latent preference process rather than a static persona: a User agent, an Assistant agent, and a Distractor agent jointly generate trajectories in which intents are partially specified, preferences emerge across turns, and noise appears at three levels (surface wording, missing information, semantic ambiguity). PPOpt adopts a reason-then-optimize paradigm: it first infers a natur

What carries the argument

Three named objects carry the argument. PersonaGym is an agentic simulator with three LLM agents—User, Assistant, and Distractor—that compiles a sampled persona into a natural-language preference specification and unrolls multi-turn query–response–feedback trajectories, with a three-layer semantic-aware noise process, producing turn-level binary outcome labels (follow-up means dissatisfaction; termination means satisfaction). PersonaAtlas is the resulting dataset of about 2,000 personas and over 10,000 conversations. PPOpt is a conditional prompt-optimizer policy that outputs an inferred user profile followed by a rewritten prompt, trained first by supervised fine-tuning on teacher-synthesiz

Load-bearing premise

The load-bearing premise is that PersonaGym's simulated users, assistants, distractors, and ground-truth synthetic personas faithfully reproduce how real people express preferences, change their minds, and respond to LLM outputs; if that simulation is unfaithful, the reported gains will not transfer to real users.

What would settle it

Run PPOpt on organic multi-turn interaction logs from real users, where personalization quality is measured by those users' own follow-up behavior or explicit ratings rather than by an LLM judge given the synthetic persona; the reported 33% personalization gain and 2.6% task-completion cost should be reproduced outside the synthetic evaluation loop.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Anyone with a frozen or proprietary LLM can improve per-user personalization by running PPOpt as an inference-time prompt rewriter, with no parameter access or gradient updates.
  • Synthetic interaction data can be generated on demand, reducing dependence on scarce, privacy-restricted real user logs and enabling flexible scaling of personalization training data.
  • PPOpt consistently improves personalization across different optimizer and deployed-model combinations, and can lift the personalization performance of a weaker model to near that of a stronger one.
  • Because the task-completion drop is small, personalization and task correctness need not trade off as sharply as they do in prior prompt-based personalization methods.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because both training and evaluation rely on the same synthetic personas and LLM-as-a-judge scoring, the reported gains partly measure internal consistency; the decisive test is whether the gains survive contact with organic, unedited user logs in a live deployment.
  • The reason-then-optimize policy could be adapted to other inference-time controls—system-prompt editing, retrieval filtering, or tool-call phrasing—wherever a fixed model must be steered per user.
  • The dataset's layered noise design (typos, dropped constraints, ambiguity) makes it possible to measure how much of PPOpt's gain comes from disentangling surface noise versus recovering missing constraints; such a decomposition is not reported and is a natural next experiment.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes two linked contributions. First, PersonaGym is a synthetic data generator that models a user as a dynamic latent preference process using three LLM agents (User, Assistant, Distractor) and releases PersonaAtlas, a large synthetic multi-turn interaction dataset. Second, PPOpt is a prompt-level, model-agnostic optimizer that, given an interaction history and a new query, first infers a natural-language user profile and then rewrites the query; it is trained with SFT followed by multi-objective RL using profile-inference and task-outcome rewards. The paper reports that PPOpt improves personalization scores over baselines while approximately preserving task-completion scores, on both synthetic and 'real-world' test sets, where the latter is built by humans editing PersonaGym-generated trajectories.

Significance. If the claims were fully validated, the contribution would be practically valuable: it offers a scalable way to obtain personalized interaction data without collecting privacy-sensitive real logs, and a black-box prompt-rewriting method applicable to frozen or proprietary LLMs. The released dataset, code, and documentation are assets to the community. The paper also introduces a more realistic treatment of personalization as a dynamic, noisy, multi-turn process rather than a static persona-preference pair, and it includes a human evaluation of the generated data. However, the central empirical claim of 'consistent improvements in personalization quality' currently rests on evaluation machinery that is coupled to the training reward, and the alleged real-world test set does not break that coupling. The results are therefore not yet convincing evidence for real-user benefit, although the underlying approach is plausible and the gap is addressable with additional validation.

major comments (3)
  1. [§4.3, Eqs. (17)–(18); §6.1; Appendix E] The training reward and the evaluation metric are both LLM judges conditioned on the ground-truth synthetic persona p_u: r_prof compares the inferred profile against p_u, r_task compares base vs. optimized responses with p_u supplied as preference grounding, and the Section 6.1 personalization score gives the user profile to the judge alongside the model output. The 'real-world' set in Appendix E is generated by editing PersonaGym trajectories, so it retains the same dependency. Consequently, the reported 33% personalization gain may measure optimization against a persona-conditioned judge rather than satisfaction of real users. The human evaluation in Table 5 validates the quality of the generated data, not whether PPOpt-optimized prompts improve personalized responses for humans. Please add a human preference evaluation of PPOpt outputs against baselines, or evaluate on real user inter
  2. [Table 2 and Appendix E] The synthetic test set overlaps with the datasets used to generate PersonaAtlas. Table 2 lists ai2_arc, MBPP, and ultrachat_200k as query sources for PersonaGym, and Appendix E uses ai2_arc, MBPP, and ultrachat_200k in the synthetic evaluation benchmark, with only IFEval and oasst1 described as held-out. This is a train/test query overlap that can inflate the reported improvements. Please report results separately for strictly held-out datasets, and ideally retrain on a non-overlapping query set.
  3. [Table 6] As printed, the Vanilla personalization-score row is exactly identical across all three deployed models (Llama-3-8b-instruct, Qwen3-8b, GPT-oss-20b) and across all five datasets: '4.620.0015.480.0135.960.0035.300.020 5.560.003' is repeated three times. This is not plausible if the models produce different responses, and it directly affects the Δ values central to Tables 6 and 7. Please verify the table and re-run the baselines; if this is a rendering error, correct it and ensure the means and variances are model-specific.
minor comments (5)
  1. [Appendix A, Algorithm 1] Lines 6 and 12 refer to 'Eq. (17)' and 'Eq. (18)' for the Noisify operations; the correct references are Eqs. (7) and (8).
  2. [§6.2 vs. Table 6] The text says the results are 'based on the deployed model of Llama-3.3-70b-instruct', but Table 6 headers list Llama-3-8b-instruct, Qwen3-8b, and GPT-oss-20b. Please clarify which models are optimizers and which are deployed models, and align the caption and text.
  3. [§4.3, Eq. (16); Appendix C] The reward weights λ_prof and λ_task appear in the objective but their values are never specified. Please report them and any tuning procedure.
  4. [§4, introduction] The phrase 'to avoid rewarding hacking' should read 'to avoid reward hacking'.
  5. [§3.5 and Abstract] The dataset is repeatedly described as 'high-fidelity' in the same sentence (e.g., 'high-fidelity synthetic dataset of high-fidelity multi-turn personalized interaction trajectories'); consider removing the duplication.

Circularity Check

2 steps flagged

PPOpt's headline personalization gains are measured by the same persona-conditioned LLM-judge convention used as its RL reward; the 'real-world' test set inherits the synthetic pipeline.

specific steps
  1. fitted input called prediction [Section 4.3, Eq. (18) (training reward) and Section 6.1, Evaluation Metrics (headline metric)]
    "(2) Task outcome reward (pairwise, preference-aware). We evaluate whether the improved prompt yields a better assistant response than the original prompt. We use an LLM-as-judge J_task that is provided with: the original query, the ground-truth persona p_u (for preference grounding), and the two candidate responses ... r_task(u) = Norm(J_task(...)) (18)"

    PPOpt is trained by RL to maximize r_task (Eq. 18), a persona-conditioned LLM judge rating the optimized response against the ground-truth persona. The paper's central evidence of 'personalization quality' (Section 6.1) is the same construction: 'we provide the user profile together with the model output to an LLM-based judge, which assigns a score indicating how well the response aligns with the specified user preferences.' Thus the reported 33.09% personalization gain measures the trained objective itself on synthetic data, not an independent user preference signal. Different judge models or splits do not break this coupling because the preference-grounding convention is identical.

  2. other [Appendix E, 'Real-world dataset construction']
    "We first generate interaction trajectories using our automated generation pipeline. ... Human experts are instructed to edit and refine the user-side content only, including the initial queries and subsequent user feedback, according to specified user profiles."

    The 'real-world' test set is a human-edited version of PersonaGym trajectories, not independent interaction logs. It therefore inherits the same personas, trajectory structure, and persona-conditioned judge convention, so Table 9 cannot validate transfer to real users; it remains on the same synthetic distribution used for training and reward computation. This appendix passage confirms, rather than breaks, the train/eval coupling.

full rationale

The paper is self-contained and does not rely on self-citation chains; no uniqueness theorem or ansatz is imported from the authors' prior work. However, the central personalization claim is coupled to its training objective by construction. Eq. (18) defines the RL reward as an LLM judge scoring response-persona alignment; Section 6.1 defines the headline personalization score the same way. The appendix's 'real-world' set is built by editing PersonaGym trajectories, so it does not provide an external benchmark. The human evaluation (Table 5, Appendix I) validates the generated data, not PPOpt's output. Task-completion scores and robustness ablations retain some independent content, but the main 'personalization quality' result is partially circular.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 1 invented entities

The paper's central claim rests on the fidelity of its synthetic personas and on LLM-judge agreement with those personas. The main hand-chosen numbers (ρ, noise strengths, reward weights, masking policy) control exactly how much preference signal and noise appear in training, and they are not tuned against external real-user data. The axioms make explicit that simulation fidelity and judge reliability are assumed rather than demonstrated.

free parameters (5)
  • stylization probability ρ = 0.5
    Eq. (3); chosen by hand to balance clean and persona-stylized queries; directly controls how much preference signal appears in training histories.
  • distractor noise strengths α_q, α_f and probabilities p_q, p_f = not reported in the provided text
    Eqs. (7)-(8) and Algorithm 1; control the amount of corruption in trajectories, yet no values or sensitivity analysis are given in the main text.
  • RL reward weights λ_prof, λ_task = not reported in the provided text
    Eq. (16); trade off profile inference quality against task outcome in the multi-objective RL objective; values are not specified in the main text.
  • feature masking policy π_mask = random masking per feature, rate unspecified
    Eq. (1); controls partial observability of personas, but the exact masking rate or schedule is not given.
  • training hyperparameters (LoRA rank, GRPO K, temperature, nucleus p) = r=32, K=8, τ=0.7, p=0.9
    Appendix C; standard hand-chosen training settings rather than fitted values, but they affect the reported optimization results.
axioms (5)
  • domain assumption The simulated User, Assistant, and Distractor agents produce interaction trajectories that faithfully mimic real user preference expression, feedback, and noise.
    Used throughout Sections 3.3-3.5; the entire training and evaluation pipeline assumes simulated conversations are realistic, supported only by internal human evaluation.
  • domain assumption Ground-truth synthetic persona p_u is a valid representation of a user's preferences and can serve as reference for rewards and evaluation.
    Eqs. (17)-(18) score r_prof and r_task against p_u; the personalization metric also gives the user profile to the judge. No independent real-user preference data validates p_u.
  • domain assumption LLM-as-judge models (GPT-4o-mini / GPT-5.2) reliably score personalization and task completion.
    Reward computation (Eqs. 17-18) and all evaluation metrics rely on J_prof and J_task; no agreement study against human judgments is reported for these scoring tasks.
  • domain assumption Outcome labels y_t = 1 - I[follow-up at t] are a valid proxy for user satisfaction.
    Eq. (6) derives positive/negative training signals purely from whether the simulated user continues; real users may continue for reasons other than dissatisfaction.
  • domain assumption The 'real-world' test set, human-edited from PersonaGym trajectories, approximates genuinely real user-LLM interactions.
    Appendix E: expert editors modify user-side content only, but trajectories originate from the same synthetic generator, so distributional independence from the training pipeline is limited.
invented entities (1)
  • Dynamic latent preference process (simulated user persona) no independent evidence
    purpose: Ground truth for generating interaction trajectories and for reward and evaluation scoring in PPOpt.
    The persona feature-value sets are constructed by the authors; no external data links them to real users' latent preferences. Human evaluation is internal to the paper and the dataset is released, but it is not independent evidence of real-world fidelity.

pith-pipeline@v1.3.0-alltime-deepseek · 27114 in / 13484 out tokens · 114645 ms · 2026-08-02T23:48:37.618724+00:00 · methodology

0 comments
read the original abstract

Personalized prompting offers large opportunities for deploying large language models (LLMs) to diverse users, yet existing prompt optimization methods primarily focus on task-level optimization while largely overlooking user-specific preferences and latent constraints of individual users. This gap is primarily due to (i) the absence of high-quality, privacy-sensitive data that capture personalized user-LLM interactions at scale, and (ii) the lack of robust reward signals for individual preferences. To overcome existing data limitations, we introduce a high-fidelity synthetic data generation framework called PersonaGym. Unlike prior work that treats personalization as static persona-preference pairs, PersonaGym models a dynamic preference process via an agentic LLM system to simulate realistic preference behaviors and semantic-aware noise in order to generate personalized multi-turn interaction trajectories. Using PersonaGym, we release PersonaAtlas, a large-scale, high-quality, and diverse synthetic dataset of high-fidelity multi-turn personalized interaction trajectories that closely mirror real-world preference expression and noise patterns. We further propose Personalized Prompt Optimization (PPOpt), a scalable and model-agnostic framework that optimizes user prompts based on interaction histories without modifying the deployed LLM. PPOpt adopts a reason-then-optimize paradigm that infers an explicit user profile and conditions prompt rewriting on the user profile to avoid reward hacking. Our training procedure for PPOpt integrates a cold-start supervised prior with outcome-driven multi-objective reinforcement learning. We present extensive experiments to demonstrate consistent improvements over state-of-the-art baselines in terms of task performance, personalization quality, and robustness to noisy as well as to sparse preference signals.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Post-training makes large language models less human-like

    cs.CL 2026-05 unverdicted novelty 6.0

    Post-training reduces LLMs' behavioral alignment with humans across families and sizes, with the misalignment increasing in newer generations while persona induction fails to improve individual-level predictions.

  2. Guardian-as-an-Advisor: Advancing Next-Generation Guardian Models for Trustworthy LLMs

    cs.LG 2026-04 unverdicted novelty 5.0

    Guardian-as-an-Advisor prepends risk labels and explanations from a guardian model to queries, improving LLM safety compliance and reducing over-refusal while adding minimal compute overhead.

Reference graph

Works this paper leans on

104 extracted references · 31 linked inside Pith · cited by 2 Pith papers

  1. [1]

    AI@Meta. 2024. Llama 3 Model Card. (2024). https://github.com/meta-llama/ llama3/blob/main/MODEL_CARD.md

    2024

  2. [2]

    Anthropic. 2025. Claude Sonnet 4.5. https://www.anthropic.com/claude/sonnet Accessed: 2026-02-06

    2025

  3. [3]

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021. Program synthesis with large language models.arXiv preprint arXiv:2108.07732(2021)

    Pith/arXiv arXiv 2021

  4. [4]

    Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. 2022. Constitutional AI: Harmlessness from AI feedback.arXiv preprint arXiv:2212.08073(2022)

    Pith/arXiv arXiv 2022

  5. [5]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners.Advances in Neural Information Processing Systems33 (2020), 1877–1901

    2020

  6. [6]

    Hyungjune Bu, Chanjoo Jung, Minjae Kang, and Jaehyung Kim. 2025. Personalized LLM Decoding via Contrasting Personal Preference.arXiv preprintarXiv:2506.12109 (2025). 15 Synthetic Interaction Data for Scalable Personalization in Large Language Models

    arXiv 2025

  7. [7]

    Xin Chan, Xiaoyang Wang, Dian Yu, Haitao Mi, and Dong Yu. 2024. Scaling Synthetic Data Creation with 1,000,000,000 Personas.arXiv preprintarXiv:2406.20094 (2024)

    Pith/arXiv arXiv 2024

  8. [8]

    Ruizhe Chen, Xiaotian Zhang, Meng Luo, Wenhao Chai, and Zuozhu Liu. 2024. PAD: Personal- ized Alignment of LLMs at Decoding-time. InInternational Conference on Learning Representa- tions

    2024

  9. [9]

    Jiale Cheng, Xiao Liu, Kehan Zheng, Pei Ke, Hongning Wang, Yuxiao Dong, Jie Tang, and Minlie Huang. 2024. Black-box prompt optimization: Aligning large language models without model training. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 3201–3219

    2024

  10. [10]

    John Joon Young Chung, Ece Kamar, and Saleema Amershi. 2023. Increasing diversity while maintaining accuracy: Text data generation with large language models and human interven- tions.arXiv preprint arXiv:2306.04140(2023)

    Pith/arXiv arXiv 2023

  11. [11]

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge.arXiv:1803.05457v1(2018)

    Pith/arXiv arXiv 2018

  12. [12]

    Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. 2023. Free Dolly: Introducing the World’s First Truly Open Instruction-Tuned LLM

    2023

  13. [13]

    Wendi Cui, Jiaxin Zhang, Zhuohang Li, Hao Sun, Damien Lopez, Kamalika Das, Bradley Malin, and Kumar Sricharan. 2024. SEE: Strategic Exploration and Exploitation for Cohesive In-Context Prompt Optimization.https://api.semanticscholar.org/CorpusID:267750577

    2024

  14. [14]

    Haixing Dai, Zhengliang Liu, Wenxiong Liao, Xiaoke Huang, Yihan Cao, Zihao Wu, Lin Zhao, Shaochen Xu, Fang Zeng, Wei Liu, et al. 2025. Auggpt: Leveraging chatgpt for text data augmentation.IEEE Transactions on Big Data(2025)

    2025

  15. [15]

    Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. 2023. Enhancing chat language models by scaling high-quality instructional conversations. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 3029–3051

    2023

  16. [16]

    Yin Fang, Xiaozhuan Liang, Ningyu Zhang, Kangwei Liu, Rui Huang, Zhuo Chen, Xiaohui Fan, and Huajun Chen. 2024. Mol-Instructions: A Large-Scale Biomolecular Instruction Dataset for Large Language Models. InICLR. OpenReview.net. https://openreview.net/pdf?id= Tlsdsb6l9n

    2024

  17. [17]

    Shuzheng Gao, Chaozheng Wang, Cuiyun Gao, Xiaoqian Jiao, Chun Yong Chong, Shan Gao, and Michael R. Lyu. 2025. The Prompt Alchemist: Automated LLM-Tailored Prompt Optimization for Test Case Generation.arXiv preprintarXiv:2501.01329 (2025)

    Pith/arXiv arXiv 2025

  18. [18]

    Cristina Garbacea and Chenhao Tan. 2025. HyPerAlign: Interpretable Personalized LLM Alignment via Hypothesis Generation.arXiv preprintarXiv:2505.00038 (2025)

    Pith/arXiv arXiv 2025

  19. [19]

    Jian Guan, Jun Wu, Jia-Nan Li, Chuanqi Cheng, and Wei Wu. 2025. A Survey on Personalized Alignment - The Missing Piece for Large Language Models in Real-World Applications.arXiv preprintarXiv:2503.17003 (2025). 16 Synthetic Interaction Data for Scalable Personalization in Large Language Models

    Pith/arXiv arXiv 2025

  20. [20]

    Qingyan Guo, Rui Wang, Junliang Guo, Bei Li, Kaitao Song, Xu Tan, Guoqing Liu, Jiang Bian, Yujiu Yang, Tsinghua University, and Microsoft Research. 2023. EvoPrompt: Connecting LLMs with Evolutionary Algorithms Yields Powerful Prompt Optimizers

    2023

  21. [21]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring Massive Multitask Language Understanding.Proceedings of the International Conference on Learning Representations (ICLR)(2021)

    2021

  22. [22]

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. 2022. Lora: Low-rank adaptation of large language models.ICLR1, 2 (2022), 3

    2022

  23. [23]

    Yue Huang, Zhengzhe Jiang, Xiaonan Luo, Kehan Guo, Haomin Zhuang, Yujun Zhou, Zhengqing Yuan, Xiaoqi Sun, Jules Schleinitz, Yanbo Wang, et al. 2025. ChemOrch: Empowering LLMs with Chemical Intelligence via Synthetic Instructions.arXiv preprint arXiv:2509.16543(2025)

    arXiv 2025

  24. [24]

    Yue Huang, Siyuan Wu, Chujie Gao, Dongping Chen, Qihui Zhang, Yao Wan, Tianyi Zhou, Chaowei Xiao, Jianfeng Gao, Lichao Sun, and Xiangliang Zhang. 2025. DataGen: Unified Syn- thetic Dataset Generation via Large Language Models. InThe Thirteenth International Conference on Learning Representations.https://openreview.net/forum?id=F5R0lG74Tu

    2025

  25. [25]

    Bowen Jiang, Yuan Yuan, Maohao Shen, Zhuoqun Hao, Zhangchen Xu, Zichen Chen, Ziyi Liu, Anvesh Rao Vijjini, Jiashu He, Hanchao Yu, Radha Poovendran, Gregory Wornell, Lyle Ungar, Dan Roth, Sihao Chen, and Camillo Jose Taylor. 2025. PersonaMem-v2: Towards Personalized Intelligence via Learning Implicit User Personas and Agentic Memory. arXiv:2512.06688 [cs.CL]

    arXiv 2025

  26. [26]

    Tae Soo Kim, Yoonjoo Lee, Yoonah Park, Jiho Kim, Young-Ho Kim, and Juho Kim. 2025. CUPID: Evaluating Personalized and Contextualized Alignment of LLMs from Interactions.arXiv preprint arXiv:2508.01674 (2025)

    Pith/arXiv arXiv 2025

  27. [27]

    AndreasKöpf, YannicKilcher, DimitriVonRütte, SotirisAnagnostidis, ZhiRuiTam, KeithStevens, Abdullah Barhoum, Duc Nguyen, Oliver Stanley, Richárd Nagyfi, et al. 2023. Openassistant conversations-democratizing large language model alignment.Advances in neural information processing systems36 (2023), 47669–47681

    2023

  28. [28]

    Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, et al. 2023. Rlaif vs. rlhf: Scaling reinforcementlearningfromhumanfeedbackwithaifeedback.arXivpreprintarXiv:2309.00267

    Pith/arXiv arXiv 2023

  29. [29]

    Seongyun Lee, Sue Hyun Park, Seungone Kim, and Minjoon Seo. 2024. Aligning to thousands of preferences via system message generalization.Advances in Neural Information Processing Systems37 (2024), 73783–73829

    2024

  30. [30]

    Changhao Li, Yuchen Zhuang, Rushi Qiang, Haotian Sun, Hanjun Dai, Chao Zhang, and Bo Dai

  31. [31]

    Jia-Nan Li, Jian Guan, Songhao Wu, Wei Wu, and Rui Yan. 2025. From 1,000,000 Users to Every User: Scaling Up Personalized Preference for User-level Alignment.arXiv preprint arXiv:2503.15463 (2025)

    Pith/arXiv arXiv 2025

  32. [32]

    Junyong Lin, Lu Dai, Ruiqian Han, Yijie Sui, Ruilin Wang, Xingliang Sun, Qinglin Wu, Min Feng, Hao Liu, and Hui Xiong. 2025. ScIRGen: Synthesize Realistic and Large-Scale RAG Dataset for Scientific Research. InProceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2(Toronto ON, Canada)(KDD ’25). Association for Computing Ma...

    arXiv 2025

  33. [33]

    Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. Truthfulqa: Measuring how models mimic human falsehoods. InProceedings of the 60th annual meeting of the association for computational linguistics (volume 1: long papers). 3214–3252

    2022

  34. [34]

    Ruibo Liu, Jerry Wei, Fangyu Liu, Chenglei Si, Yanzhe Zhang, Jinmeng Rao, Steven Zheng, Daiyi Peng, Diyi Yang, Denny Zhou, et al. 2024. Best practices and lessons learned on synthetic data.arXiv preprint arXiv:2404.07503(2024)

    Pith/arXiv arXiv 2024

  35. [35]

    Anton Lozhkov, Hynek Kydlíček, Loubna Ben Allal, Guilherme Penedo, Edward Beeching, Quentin Gallouédec, Nathan Habib, Lewis Tunstall, and Leandro von Werra. 2025. OpenR1- Math-220k.https://huggingface.co/datasets/open-r1/OpenR1-Math-220k

    2025

  36. [36]

    OpenAI. 2024. Hello GPT-4o. https://openai.com/index/hello-gpt-4o/ Accessed: 2026-02-06

    2024

  37. [37]

    OpenAI. 2025. gpt-oss-120b & gpt-oss-20b Model Card. arXiv:2508.10925 [cs.CL]https: //arxiv.org/arXiv:2508.10925

    Pith/arXiv arXiv 2025

  38. [38]

    OpenAI. 2025. Introducing GPT-5.2. https://openai.com/index/ introducing-gpt-5-2/Accessed: 2026-02-06

    2025

  39. [39]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. InAdvances in Neural Information Processing Systems

    2022

  40. [40]

    Silviu Pitis, Ziang Xiao, Nicolas Le Roux, and Alessandro Sordoni. 2024. Improving Context- Aware Preference Modeling for Language Models.arXiv preprintarXiv:2407.14916 (2024)

    Pith/arXiv arXiv 2024

  41. [41]

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. (2023)

    2023

  42. [42]

    Arij Riabi, Thomas Scialom, Rachel Keraron, Benoît Sagot, Djamé Seddah, and Jacopo Sta- iano. 2021. Synthetic Data Augmentation for Zero-Shot Cross-Lingual Question Answering. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Associ- ation for Computational Linguistics, Online and Punta Cana, Dominican Republic, 70...

    2021

  43. [43]

    Haris Riaz, Sourav Bhabesh, Vinayak Arannil, Miguel Ballesteros, and Graham Horwood. 2025. MetaSynth: Meta-Prompting-Driven Agentic Scaffolds for Diverse Synthetic Data Generation. arXiv preprint arXiv:2504.12563(2025)

    arXiv 2025

  44. [44]

    IntegratingSummarizationandRetrievalforEnhanced Personalization via Large Language Models.arXiv preprintarXiv:2310.20081 (2023)

    Chris Richardson, Yao Zhang, Kellen Gillespie, Sudipta Kar, Arshdeep Singh, Zeynab Raeesy, OmarZiaKhan,andAbhinavSethy.2023. IntegratingSummarizationandRetrievalforEnhanced Personalization via Large Language Models.arXiv preprintarXiv:2310.20081 (2023)

    Pith/arXiv arXiv 2023

  45. [45]

    Caitlin M Rivers and Bryan L Lewis. 2014. Ethical research standards in a world of big data. F1000Research3 (2014), 38

    2014

  46. [46]

    Alireza Salemi, Sheshera Mysore, Michael Bendersky, and Hamed Zamani. 2024. LaMP: When Large Language Models Meet Personalization. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Compu- tational Linguistics, Bangkok, Thailand, 7370–7392.https://aclanthology.org/2024. acl-lo...

    2024

  47. [47]

    Timo Schick and Hinrich Schütze. 2021. Generating Datasets with Pretrained Language Models. InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Online and Punta Cana, Dominican Republic, 6943– 6951.https://aclanthology.org/2021.emnlp-main.555/

    2021

  48. [48]

    ShareGPT. 2023. ShareGPT.https://sharegpt.com/

    2023

  49. [49]

    Claudio Spiess, Mandana Vaziri, Louis Mandel, and Martin Hirzel. 2025. AutoPDL: Automatic Prompt Optimization for LLM Agents.arXiv preprintarXiv:2504.04365 (2025)

    arXiv 2025

  50. [50]

    Zhaoxuan Tan, Zheng Li, Tianyi Liu, Haodong Wang, Hyokun Yun, Ming Zeng, Pei Chen, Zhihan Zhang, Yifan Gao, Ruijie Wang, Priyanka Nigam, Bing Yin, and Menghan Jiang. 2025. Aligning Large Language Models with Implicit Preferences from User-Generated Content.arXiv preprint arXiv:2506.04463 (2025)

    Pith/arXiv arXiv 2025

  51. [51]

    Zhaoxuan Tan, Zheyuan Liu, and Meng Jiang. 2024. Personalized Pieces: Efficient Personalized Large Language Models through Collaborative Efforts. InConference on Empirical Methods in Natural Language Processing

    2024

  52. [52]

    Zhaoxuan Tan, Zixuan Zhang, Haoyang Wen, Zheng Li, Rongzhi Zhang, Pei Chen, Fengran Mo, Zheyuan Liu, Qingkai Zeng, Qingyu Yin, and Menghan Jiang. 2025. Instant Personalized Large Language Model Adaptation via Hypernetwork.arXiv preprintarXiv:2510.16282 (2025)

    Pith/arXiv arXiv 2025

  53. [53]

    Hashimoto

    Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford Alpaca: An Instruction-following LLaMA model.https://github.com/tatsu-lab/stanford_alpaca

    2023

  54. [54]

    Qwen Team. 2025. Qwen3 Technical Report. arXiv:2505.09388 [cs.CL]https://arxiv. org/arXiv:2505.09388

    Pith/arXiv arXiv 2025

  55. [55]

    Sarah Tran, Hongfan Lu, Isaac Slaughter, Bernease Herman, Aayushi Dangol, Yue Fu, Lufei Chen, Biniyam Gebreyohannes, Bill Howe, Alexis Hiniker, et al. 2025. Understanding Privacy Norms Around LLM-Based Chatbots: A Contextual Integrity Perspective. InProceedings of the AAAI/ACM Conference on AI, Ethics, and Society, Vol. 8. 2522–2534

    2025

  56. [56]

    Tiannan Wang, Meiling Tao, Ruoyu Fang, Huilin Wang, Shuai Wang, Yuchen Eleanor Jiang, and Wangchunshu Zhou. 2024. AI PERSONA: Towards Life-long Personalization of LLMs.arXiv preprintarXiv:2412.13103 (2024)

    Pith/arXiv arXiv 2024

  57. [57]

    Xiangqi Wang, Yue Huang, Yanbo Wang, Xiaonan Luo, Kehan Guo, Yujun Zhou, and Xiangliang Zhang. 2025. AdaReasoner: Adaptive Reasoning Enables More Flexible Thinking. InThe Thirty- ninth Annual Conference on Neural Information Processing Systems.https://openreview. net/forum?id=VjjJlJ5qik

    2025

  58. [58]

    Yanbo Wang, Zixiang Xu, Yue Huang, Xiangqi Wang, Zirui Song, Lang Gao, Chenxi Wang, Xiangru Tang, Yue Zhao, Arman Cohan, Xiangliang Zhang, and Xiuying Chen. 2025. DyFlow: Dynamic Workflow Framework for Agentic Reasoning. InThe Thirty-ninth Annual Conference on NeuralInformationProcessingSystems. https://openreview.net/forum?id=0pbUfmwNTy

    2025

  59. [59]

    Jerry Wei, Da Huang, Yifeng Lu, Denny Zhou, and Quoc V Le. 2023. Simple synthetic data reduces sycophancy in large language models.arXiv preprint arXiv:2308.03958(2023)

    Pith/arXiv arXiv 2023

  60. [60]

    Shujin Wu, May Fung, Cheng Qian, Jeonghwan Kim, Dilek Hakkani-Tur, and Heng Ji. 2024. Aligning LLMs with Individual Preferences via Interaction.arXiv preprintarXiv:2410.03642 (2024). 19 Synthetic Interaction Data for Scalable Personalization in Large Language Models

    Pith/arXiv arXiv 2024

  61. [61]

    Shirley Wu, Michel Galley, Baolin Peng, Hao Cheng, Gavin Li, Yao Dou, Weixin Cai, James Zou, Jure Leskovec, and Jianfeng Gao. 2025. Collabllm: From passive responders to active collaborators. InInternational Conference on Machine Learning

    2025

  62. [62]

    Jinyu Xiang, Jiayi Zhang, Zhaoyang Yu, Fengwei Teng, Jinhao Tu, Xinbing Liang, Sirui Hong, Chenglin Wu, and Yuyu Luo. 2025. Self-Supervised Prompt Optimization.arXiv preprint arXiv:2502.06855 (2025)

    Pith/arXiv arXiv 2025

  63. [63]

    Zhangchen Xu, Fengqing Jiang, Luyao Niu, Yuntian Deng, Radha Poovendran, Yejin Choi, and Bill Yuchen Lin. 2024. Magpie: Alignment data synthesis from scratch by prompting aligned llms with nothing.arXiv preprint arXiv:2406.08464(2024)

    Pith/arXiv arXiv 2024

  64. [64]

    Chao Yu, Qixin Tan, Hong Lu, Jiaxuan Gao, Xinting Yang, Yu Wang, Yi Wu, and Eugene Vinitsky

  65. [65]

    Tom Zehle, Timo Heiss, Moritz Schlager, Matthias Assenmacher, and Matthias Feurer. 2025. promptolution: A Unified, Modular Framework for Prompt Optimization.arXiv preprint arXiv:2512.02840 (2025)

    arXiv 2025

  66. [66]

    ICPL: Few-shot In-context Preference Learning via LLMs

  67. [67]

    Jiayi Zhang, Jinyu Xiang, Zhaoyang Yu, Fengwei Teng, Xionghui Chen, Jiaqi Chen, Mingchen Zhuge, Xin Cheng, Sirui Hong, Jinlin Wang, Bingnan Zheng, Bangbang Liu, Yuyu Luo, and Chenglin Wu. 2024. AFlow: Automating Agentic Workflow Generation.arXiv preprint arXiv:2410.10762 (2024)

    Pith/arXiv arXiv 2024

  68. [68]

    Evaluating Large Language Models at Evaluating Instruction Following

    ZhiyuanZeng,JiatongYu,TianyuGao,YuMeng,TanyaGoyal,andDanqiChen.2024. Evaluating Large Language Models at Evaluating Instruction Following. InInternational Conference on Learning Representations (ICLR)

    2024

  69. [69]

    Linhai Zhang, Jialong Wu, Deyu Zhou, and Yulan He. 2025. PROPER: A Progressive Learning Framework for Personalized Large Language Models with Group-Level Adaptation. InAnnual Meeting of the Association for Computational Linguistics

    2025

  70. [70]

    Letian Zhang, Quan Cui, Bingchen Zhao, and Cheng Yang. 2025. Oasis: One Image is All You Need for Multimodal Instruction Data Synthesis.arXiv preprint arXiv:2503.08741(2025)

    Pith/arXiv arXiv 2025

  71. [71]

    Zhehao Zhang, Ryan A. Rossi, Branislav Kveton, Yijia Shao, Diyi Yang, Hamed Zamani, Franck Dernoncourt, Joe Barrow, Tong Yu, Sungchul Kim, Ruiyi Zhang, Jiuxiang Gu, Tyler Derr, Hongjie Chen, Ju-Ying Wu, Xiang Chen, Zichao Wang, Subrata Mitra, Nedim Lipka, Nesreen K. Ahmed, and Yu Wang. 2024. Personalization of Large Language Models: A Survey.arXiv preprin...

    Pith/arXiv arXiv 2024

  72. [72]

    Xiaotian Zhang, Ruizhe Chen, Yang Feng, and Zuozhu Liu. 2025. Persona-judge: Personalized Alignment of Large Language Models via Token-level Self-judgment. InAnnual Meeting of the Association for Computational Linguistics

    2025

  73. [73]

    Xiaoyan Zhao, Juntao You, Yang Zhang, Wenjie Wang, Hong Cheng, Fuli Feng, See-Kiong Ng, and Tat-Seng Chua. 2025. NextQuill: Causal Preference Modeling for Enhancing LLM Personalization.arXiv preprintarXiv:2506.02368 (2025). 20 Synthetic Interaction Data for Scalable Personalization in Large Language Models

    arXiv 2025

  74. [74]

    Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng. [n.d.]. WildChat: 1M ChatGPT Interaction Logs in the Wild. InThe Twelfth International Conference on Learning Representations

  75. [75]

    Tianyu Zheng, Ge Zhang, Tianhao Shen, Xueling Liu, Bill Yuchen Lin, Jie Fu, Wenhu Chen, and Xiang Yue. 2024. Opencodeinterpreter: Integrating code generation with execution and refinement.arXiv preprint arXiv:2402.14658(2024)

    Pith/arXiv arXiv 2024

  76. [76]

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena.Advances in Neural Information Processing Systems36 (2023), 46595–46623

    2023

  77. [77]

    Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. 2023. Instruction-following evaluation for large language models.arXiv preprint arXiv:2311.07911(2023)

    Pith/arXiv arXiv 2023

  78. [78]

    Ziije Zhong, Linqing Zhong, Zhaoze Sun, Qingyun Jin, Zengchang Qin, and Xiaofan Zhang

  79. [79]

    Synthet2c: Generating synthetic data for fine-tuning large language models on the text2cypher task.arXiv preprint arXiv:2406.10710(2024)

    Pith/arXiv arXiv 2024

  80. [81]

    Yuchen Zhuang, Haotian Sun, Yue Yu, Rushi Qiang, Qifan Wang, Chao Zhang, and Bo Dai. 2024. HYDRA: Model Factorization Framework for Black-Box LLM Personalization.arXiv preprint arXiv:2406.02888 (2024)

    Pith/arXiv arXiv 2024

Showing first 80 references.