Pith. sign in

REVIEW 2 major objections 1 minor 41 references

Reviewed by Pith at T0; open to challenge.

T0 means a machine referee read the full paper against a public rubric. The mark states how deep the mechanical check went, never who wrote it. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.3

MATO turns multi-objective LLM alignment into a test-time optimization task that recovers rewards from the model itself and tunes objective weights during decoding.

2026-06-29 23:00 UTC pith:UMRBJZVU

load-bearing objection MATO combines internal reward discovery with test-time weight optimization for training-free multi-objective LLM alignment, but the experiments leave the accuracy of those discovered rewards untested. the 2 major comments →

arxiv 2605.25342 v1 pith:UMRBJZVU submitted 2026-05-25 cs.CL

MATO: Multi-objective Personalized Alignment with Test-time Optimization for Large Language Models

Linhao Luo , Thuy-Trang Vu , Van-Anh Nguyen , Junae Kim , Gholamreza Haffari , Dinh Phung This is my paper
classification cs.CL
keywords multi-objective alignmenttest-time optimizationpersonalized alignmenttraining-free methodsreward discoverydecoding optimizationLLM steerabilitypreference balancing
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper establishes that personalization with conflicting user preferences can be handled without training or external reward models by casting the problem as an optimization solved while the model generates each token. A reward discovery step extracts usable signals directly from the backbone LLM for any natural-language objective, and a weight optimization step adjusts their relative strengths based on the user's stated priorities and the response so far. These two signals then steer an online search over the token distribution. Experiments across datasets and models indicate this produces better trade-offs than prompting or trained baselines and gives users more reliable control when objectives pull in different directions.

Core claim

MATO formulates personalization as a test-time optimization problem that steers the relative importance of multiple objectives through controllable weights during decoding, without modifying model parameters or requiring external reward models. A reward discovery module recovers preference rewards directly from the backbone LLM for diverse objectives specified in natural language, while a weight optimization module dynamically adjusts objective weights based on the user's initial preferences and the partially generated response to balance competing objectives during generation. The resulting rewards and weights jointly guide an online optimization procedure over the token distribution.

What carries the argument

The reward discovery module that extracts preference signals from the LLM combined with the weight optimization module that adjusts controllable weights to guide token-level optimization during decoding.

Load-bearing premise

The backbone LLM can recover accurate and useful preference rewards directly from natural-language objective descriptions without external reward models or any training.

What would settle it

Human preference ratings on responses generated under conflicting objectives would show no reliable improvement over strong prompting baselines, or the recovered rewards would fail to correlate with independent human judgments of the stated objectives.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Produces Pareto improvements in multi-objective alignment across multiple datasets and backbone LLMs.
  • Delivers stronger steerability, allowing users to set and maintain desired trade-offs between conflicting preferences.
  • Adapts to new or changing preferences without retraining or new reward models.
  • Remains model-agnostic and training-free while still outperforming methods that require either.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The approach could scale to interactive settings where preferences evolve over a conversation by re-optimizing weights at each turn.
  • Internal model knowledge may substitute for curated preference datasets in many alignment scenarios.
  • If the optimization scales efficiently, the same machinery could support finer-grained control over larger numbers of simultaneous objectives.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 1 minor

Summary. The paper proposes MATO, a training-free framework for multi-objective personalized LLM alignment via test-time optimization. It introduces a reward discovery module that extracts scalar preference rewards directly from a frozen backbone LLM for natural-language objectives, a weight optimization module that dynamically balances competing objectives based on user preferences and partial generations, and an online optimization procedure over the token distribution during decoding. The central claims are that MATO achieves Pareto-improving alignment and stronger steerability compared to strong baselines across multiple datasets and backbone models, without requiring parameter updates or external reward models.

Significance. If the reward discovery module produces faithful, calibrated rewards for arbitrary natural-language objectives, the approach would offer a scalable, model-agnostic alternative to training-based or RM-dependent multi-objective alignment methods, potentially improving controllability when objectives conflict. The test-time optimization framing is a clear strength if the recovered rewards are shown to be reliable.

major comments (2)
  1. [methods / reward discovery] Reward discovery module (methods section): the claim that accurate preference rewards can be recovered directly from the frozen backbone LLM for arbitrary natural-language objectives is load-bearing for the training-free and Pareto-improvement assertions, yet no direct validation (e.g., correlation with human preference labels, comparison to external reward models, or ablation on reward quality) is reported; only aggregate end-to-end metrics are shown.
  2. [experiments] Experiments section: the abstract and results claim consistent outperformance and Pareto improvement, but without reported quantitative tables, baseline details, or ablations on the reward discovery step, it is impossible to assess whether gains stem from faithful rewards or from other components of the online optimization.
minor comments (1)
  1. [abstract] Abstract lacks any numerical results, baseline names, or dataset details, which weakens the ability to evaluate the strength of the empirical claims at first reading.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback highlighting the importance of validating the reward discovery module and clarifying experimental details. We address each major comment below and commit to revisions that strengthen the manuscript without altering its core claims.

read point-by-point responses

  1. Referee: [methods / reward discovery] Reward discovery module (methods section): the claim that accurate preference rewards can be recovered directly from the frozen backbone LLM for arbitrary natural-language objectives is load-bearing for the training-free and Pareto-improvement assertions, yet no direct validation (e.g., correlation with human preference labels, comparison to external reward models, or ablation on reward quality) is reported; only aggregate end-to-end metrics are shown.

    Authors: We agree that direct validation of reward quality would provide additional support for the load-bearing claim. Our design choice emphasized end-to-end alignment metrics because the practical value of recovered rewards lies in their effect on generation rather than isolated correlation scores, and arbitrary natural-language objectives lack standardized human labels. Nevertheless, we will add a new subsection in the revised manuscript that includes comparisons of recovered rewards to external reward models on applicable preference datasets and reports any available correlations with human labels to better isolate the module's contribution. revision: yes

  2. Referee: [experiments] Experiments section: the abstract and results claim consistent outperformance and Pareto improvement, but without reported quantitative tables, baseline details, or ablations on the reward discovery step, it is impossible to assess whether gains stem from faithful rewards or from other components of the online optimization.

    Authors: The manuscript already contains quantitative tables (Tables 1–3) and baseline descriptions in Section 4. We acknowledge, however, that explicit ablations focused on the reward discovery step are absent and would help readers attribute performance gains. In revision we will insert an ablation study that disables or substitutes the reward discovery module (e.g., uniform weights or external RM baselines) while keeping the remainder of the pipeline fixed, thereby clarifying the module's role in the observed Pareto improvements. revision: partial

Circularity Check

0 steps flagged

No circularity in derivation chain; method is a proposed procedure validated by experiments.

full rationale

The paper introduces MATO as a training-free test-time optimization framework consisting of a reward discovery module and weight optimization module. No equations, fitting procedures, or self-citations are described in the provided text that reduce any claimed prediction or result to its own inputs by construction. The central claims rest on experimental outperformance rather than a closed mathematical derivation. This is the expected non-finding for a methods paper without load-bearing self-referential reductions.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract-only; no explicit free parameters, axioms, or invented entities are stated. The central claim rests on the unstated premise that the backbone LLM can serve as a reliable internal reward source.

pith-pipeline@v0.9.1-grok · 5817 in / 1040 out tokens · 26672 ms · 2026-06-29T23:00:53.234023+00:00 · methodology

0 comments
read the original abstract

Aligning large language models (LLMs) with diverse and multifaceted user preferences is a fundamental challenge in personalized AI systems. Existing multi-objective alignment methods either rely on costly training or require pre-trained reward models for each preference, making it difficult for them to adapt to evolving preferences. Prompt-based personalization offers a training-free alternative, but prompting alone often provides limited steerability, as LLMs may overemphasize or overlook certain preferences and fail to give users reliable control over the relative importance of different objectives when conflicts arise, leading to suboptimal alignment. In this paper, we introduce MATO, a training-free framework for Multi-objective personalized Alignment with Test-time Optimization. MATO formulates personalization as a test-time optimization problem that steers the relative importance of multiple objectives through controllable weights during decoding, without modifying model parameters or requiring external reward models. Specifically, a reward discovery module recovers preference rewards directly from the backbone LLM for diverse objectives specified in natural language, while a weight optimization module dynamically adjusts objective weights based on the user's initial preferences and the partially generated response to balance competing objectives during generation. The resulting rewards and weights jointly guide an online optimization procedure over the token distribution, enabling better alignment with the target objectives. Extensive experiments across multiple datasets and backbone LLMs show that MATO consistently outperforms strong baselines, achieving Pareto-improving multi-objective alignment and stronger steerability. These results highlight test-time optimization as a promising direction for scalable, controllable, and model-agnostic personalized alignment.

Figures

Figures reproduced from arXiv: 2605.25342 by Dinh Phung, Gholamreza Haffari, Junae Kim, Linhao Luo, Thuy-Trang Vu, Van-Anh Nguyen.

Figure 1
Figure 1. Figure 1: Illustration of multi-objective personalized alignment and its challenges. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of the MATO framework, which consists of three key components: (1) reward discovery, which recovers preference rewards from the backbone LLM, (2) weight optimization, which dynamically adjusts objective weights to balance preferences during generation, and (3) online optimization, which guides the output distribution using the learned rewards and weights. Inspired by the recent success of prompt-b… view at source ↗
Figure 3
Figure 3. Figure 3: Average preference score across dimen￾sions. 2 4 6 8 10 Number of Preferences (K) 2.25 2.50 2.75 3.00 3.25 3.50 3.75 4.00 4.25 APS Preference Prompt Amulet OPAD MATO (Ours) [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 5
Figure 5. Figure 5: Empirical Pareto fronts of different methods: † and ‡ denote training-based and training-free meth￾ods, respectively. To evaluate steerability, we vary the helpfulness-humor weight from 1 to 0 with a step size of 0.2, compute the reward for each objective, and plot the empirical Pareto front [25]. For training-based methods, we train separate models follow￾ing their original algorithms. For training-free b… view at source ↗
Figure 7
Figure 7. Figure 7: Analysis of weight optimization temperature τ and optimization steps T. Parameter Analysis. We analyze the sensitivity of MATO to two key parameters: the weight optimiza￾tion temperature τ and the number of online opti￾mization steps T. As shown in [PITH_FULL_IMAGE:figures/full_fig_p009_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Parameter analysis of α and λ. 0 10 30 60 80 100 120 150 T 0.12 0.16 0.20 Seconds / token 2 4 6 8 10 K 0.20 0.40 0.60 [PITH_FULL_IMAGE:figures/full_fig_p022_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Latency analysis with respect to the number of online optimization steps [PITH_FULL_IMAGE:figures/full_fig_p022_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Rewards and weights dynamics for informativeness ( [PITH_FULL_IMAGE:figures/full_fig_p025_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Rewards and weights dynamics for evidence ( [PITH_FULL_IMAGE:figures/full_fig_p025_11.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

41 extracted references · 2 canonical work pages · 2 internal anchors

  1. [1]

    Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback.arXiv preprint arXiv:2204.05862, 2022

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  2. [2]

    Multi- objective reinforcement learning with max-min criterion: A game-theoretic approach

    Woohyeon Byeon, Giseung Park, Jongseong Chae, Amir Leshem, and Youngchul Sung. Multi- objective reinforcement learning with max-min criterion: A game-theoretic approach. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025

    2025

  3. [3]

    When large language models meet personalization: Perspectives of challenges and opportunities.World wide web, 27(4):42, 2024

    Jin Chen, Zheng Liu, Xu Huang, Chenwang Wu, Qi Liu, Gangwei Jiang, Yuanhao Pu, Yuxuan Lei, Xiaolong Chen, Xingmei Wang, et al. When large language models meet personalization: Perspectives of challenges and opportunities.World wide web, 27(4):42, 2024

    2024

  4. [4]

    Pad: Personalized alignment of llms at decoding-time

    Ruizhe Chen, Xiaotian Zhang, Meng Luo, Wenhao Chai, and Zuozhu Liu. Pad: Personalized alignment of llms at decoding-time. InThe Thirteenth International Conference on Learning Representations, 2025

    2025

  5. [5]

    Towards next-generation intelligent assistants leveraging llm techniques

    Xin Luna Dong, Seungwhan Moon, Yifan Ethan Xu, Kshitiz Malik, and Zhou Yu. Towards next-generation intelligent assistants leveraging llm techniques. InProceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 5792–5793, 2023

    2023

  6. [6]

    Linear alignment: A closed-form solution for aligning human preferences without tuning and feedback

    Songyang Gao, Qiming Ge, Wei Shen, Shihan Dou, Junjie Ye, Xiao Wang, Rui Zheng, Yicheng Zou, Zhi Chen, Hang Yan, et al. Linear alignment: A closed-form solution for aligning human preferences without tuning and feedback. InInternational Conference on Machine Learning, pages 14702–14722. PMLR, 2024

    2024

  7. [7]

    A survey on personalized alignment—the missing piece for large language models in real-world applications

    Jian Guan, Junfei Wu, Jia-Nan Li, Chuanqi Cheng, and Wei Wu. A survey on personalized alignment—the missing piece for large language models in real-world applications. InFindings of the Association for Computational Linguistics: ACL 2025, pages 5313–5333, 2025

    2025

  8. [8]

    Context steering: Controllable personalization at inference time

    Jerry Zhi-Yang He, Sashrika Pandey, Mariah L Schrum, and Anca Dragan. Context steering: Controllable personalization at inference time. InThe Thirteenth International Conference on Learning Representations, 2025

    2025

  9. [9]

    One-shot safety alignment for large language models via optimal dualization.Advances in Neural Information Processing Systems, 37:84350–84383, 2024

    Xinmeng Huang, Shuo Li, Edgar Dobriban, Osbert Bastani, Hamed Hassani, and Dongsheng Ding. One-shot safety alignment for large language models via optimal dualization.Advances in Neural Information Processing Systems, 37:84350–84383, 2024

    2024

  10. [10]

    Aligner: Efficient alignment by learning to correct.Advances in Neural Information Processing Systems, 37:90853–90890, 2024

    Jiaming Ji, Boyuan Chen, Hantao Lou, Donghai Hong, Borong Zhang, Xuehai Pan, Juntao Dai, Tianyi Qiu, and Yaodong Yang. Aligner: Efficient alignment by learning to correct.Advances in Neural Information Processing Systems, 37:90853–90890, 2024

    2024

  11. [11]

    A survey on large language models for code generation.ACM Transactions on Software Engineering and Methodology, 35 (2):1–72, 2026

    Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. A survey on large language models for code generation.ACM Transactions on Software Engineering and Methodology, 35 (2):1–72, 2026

    2026

  12. [12]

    Aligning to thousands of preferences via system message generalization.Advances in Neural Information Processing Systems, 37:73783–73829, 2024

    Seongyun Lee, Sue Hyun Park, Seungone Kim, and Minjoon Seo. Aligning to thousands of preferences via system message generalization.Advances in Neural Information Processing Systems, 37:73783–73829, 2024

    2024

  13. [13]

    Deep reinforcement learning for multiobjective opti- mization.IEEE transactions on cybernetics, 51(6):3103–3114, 2020

    Kaiwen Li, Tao Zhang, and Rui Wang. Deep reinforcement learning for multiobjective opti- mization.IEEE transactions on cybernetics, 51(6):3103–3114, 2020

    2020

  14. [14]

    Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Security

    Yuanchun Li, Hao Wen, Weijun Wang, Xiangyu Li, Yizhen Yuan, Guohong Liu, Jiacheng Liu, Wenxing Xu, Xiang Wang, Yi Sun, et al. Personal llm agents: Insights and survey about the capability, efficiency and security.arXiv preprint arXiv:2401.05459, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  15. [15]

    Parm: Multi- objective test-time alignment via preference-aware autoregressive reward model

    Baijiong Lin, Weisen Jiang, Yuancheng Xu, Hao Chen, and Ying-Cong Chen. Parm: Multi- objective test-time alignment via preference-aware autoregressive reward model. InInterna- tional Conference on Machine Learning, pages 37874–37888. PMLR, 2025. 10

    2025

  16. [16]

    The unlocking spell on base llms: Rethinking alignment via in-context learning

    Bill Yuchen Lin, Abhilasha Ravichander, Ximing Lu, Nouha Dziri, Melanie Sclar, Khyathi Chandu, Chandra Bhagavatula, and Yejin Choi. The unlocking spell on base llms: Rethinking alignment via in-context learning. InThe Twelfth International Conference on Learning Representations, 2024

    2024

  17. [17]

    Decoding-time realignment of language models

    Tianlin Liu, Shangmin Guo, Leonardo Bianco, Daniele Calandriello, Quentin Berthet, Felipe Llinares, Jessica Hoffmann, Lucas Dixon, Michal Valko, and Mathieu Blondel. Decoding-time realignment of language models. InProceedings of the 41st International Conference on Machine Learning, pages 31015–31031, 2024

    2024

  18. [18]

    Follow-the-regularized-leader and mirror descent: Equivalence theorems and l1 regularization

    Brendan McMahan. Follow-the-regularized-leader and mirror descent: Equivalence theorems and l1 regularization. InProceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, pages 525–533. JMLR Workshop and Conference Proceedings, 2011

    2011

  19. [19]

    Chatgpt and large language models in academia: opportunities and challenges.BioData mining, 16(1):20, 2023

    Jesse G Meyer, Ryan J Urbanowicz, Patrick CN Martin, Karen O’Connor, Ruowang Li, Pei-Chen Peng, Tiffani J Bright, Nicholas Tatonetti, Kyoung Jae Won, Graciela Gonzalez-Hernandez, et al. Chatgpt and large language models in academia: opportunities and challenges.BioData mining, 16(1):20, 2023

    2023

  20. [20]

    Training language models to follow instructions with human feedback.Advances in neural information processing systems, 35:27730–27744, 2022

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback.Advances in neural information processing systems, 35:27730–27744, 2022

    2022

  21. [21]

    The max-min formulation of multi-objective reinforcement learning: from theory to a model-free algorithm

    Giseung Park, Woohyeon Byeon, Seongmin Kim, Elad Havakuk, Amir Leshem, and Youngchul Sung. The max-min formulation of multi-objective reinforcement learning: from theory to a model-free algorithm. InProceedings of the 41st International Conference on Machine Learning, pages 39616–39642, 2024

    2024

  22. [22]

    Personal- izing reinforcement learning from human feedback with variational preference learning

    Sriyash Poddar, Yanming Wan, Hamish Ivison, Abhishek Gupta, and Natasha Jaques. Personal- izing reinforcement learning from human feedback with variational preference learning. InThe Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    2024

  23. [23]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in neural information processing systems, 36:53728–53741, 2023

    2023

  24. [24]

    From $r$ to $q^*$: Your language model is secretly a q-function

    Rafael Rafailov, Joey Hejna, Ryan Park, and Chelsea Finn. From $r$ to $q^*$: Your language model is secretly a q-function. InFirst Conference on Language Modeling, 2024. URL https://openreview.net/forum?id=kEVcNxtqXk

    2024

  25. [25]

    Rewarded soups: towards pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards.Advances in Neural Information Processing Systems, 36:71095–71134, 2023

    Alexandre Rame, Guillaume Couairon, Corentin Dancette, Jean-Baptiste Gaya, Mustafa Shukor, Laure Soulier, and Matthieu Cord. Rewarded soups: towards pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards.Advances in Neural Information Processing Systems, 36:71095–71134, 2023

    2023

  26. [26]

    Decoding-time language model alignment with multiple objectives.Advances in Neural Information Processing Systems, 37:48875–48920, 2024

    Ruizhe Shi, Yifang Chen, Yushi Hu, Alisa Liu, Hannaneh Hajishirzi, Noah A Smith, and Simon S Du. Decoding-time language model alignment with multiple objectives.Advances in Neural Information Processing Systems, 37:48875–48920, 2024

    2024

  27. [27]

    Robust multi-objective controlled decoding of large language models

    Seongho Son, William Bankes, Sangwoong Yoon, Shyam Sundhar Ramesh, Xiaohang Tang, and Ilija Bogunovic. Robust multi-objective controlled decoding of large language models. ICLR, 2026

    2026

  28. [28]

    Llms as writing assistants: Exploring perspectives on sense of ownership and reasoning

    Azmine Toushik Wasi, Mst Rafia Islam, and Raima Islam. Llms as writing assistants: Exploring perspectives on sense of ownership and reasoning. InProceedings of the Third Workshop on Intelligent and Interactive Writing Assistants, pages 38–42, 2024

    2024

  29. [29]

    Genarm: Reward guided generation with autoregressive reward model for test-time alignment

    Yuancheng Xu, Udari Madhushani Sehwag, Alec Koppel, Sicheng Zhu, Bang An, Furong Huang, and Sumitra Ganesh. Genarm: Reward guided generation with autoregressive reward model for test-time alignment. InThe Thirteenth International Conference on Learning Representations, 2025. 11

    2025

  30. [30]

    Metaaligner: Towards generalizable multi-objective alignment of language models.Advances in Neural Information Processing Systems, 37:34453–34486, 2024

    Kailai Yang, Zhiwei Liu, Qianqian Xie, Jimin Huang, Tianlin Zhang, and Sophia Ananiadou. Metaaligner: Towards generalizable multi-objective alignment of language models.Advances in Neural Information Processing Systems, 37:34453–34486, 2024

    2024

  31. [31]

    Rewards-in-context: Multi-objective alignment of foundation models with dynamic preference adjustment

    Rui Yang, Xiaoman Pan, Feng Luo, Shuang Qiu, Han Zhong, Dong Yu, and Jianshu Chen. Rewards-in-context: Multi-objective alignment of foundation models with dynamic preference adjustment. InInternational Conference on Machine Learning, pages 56276–56297. PMLR, 2024

    2024

  32. [32]

    Amulet: Realignment during test time for personalized preference adaptation of llms

    Zhaowei Zhang, Fengshuo Bai, Qizhi Chen, Chengdong Ma, Mingzhi Wang, Haoran Sun, Zilong Zheng, and Yaodong Yang. Amulet: Realignment during test time for personalized preference adaptation of llms. InThe Thirteenth International Conference on Learning Representations, 2025

    2025

  33. [33]

    Personalization of large language models: A survey.Transactions on Machine Learning Research, 2025

    Zhehao Zhang, Ryan A Rossi, Branislav Kveton, Yijia Shao, Diyi Yang, Hamed Zamani, Franck Dernoncourt, Joe Barrow, Tong Yu, Sungchul Kim, et al. Personalization of large language models: A survey.Transactions on Machine Learning Research, 2025

    2025

  34. [34]

    Do llms recognize your preferences? evaluating personalized preference following in llms

    Siyan Zhao, Mingyi Hong, Yang Liu, Devamanyu Hazarika, and Kaixiang Lin. Do llms recognize your preferences? evaluating personalized preference following in llms. InThe Thirteenth International Conference on Learning Representations, 2025

    2025

  35. [35]

    Panacea: Pareto alignment via preference adaptation for llms

    Yifan Zhong, Chengdong Ma, Xiaoyuan Zhang, Ziran Yang, Haojun Chen, Qingfu Zhang, Siyuan Qi, and Yaodong Yang. Panacea: Pareto alignment via preference adaptation for llms. Advances in Neural Information Processing Systems, 37:75522–75558, 2024

    2024

  36. [36]

    Beyond one-preference-fits-all alignment: Multi-objective direct preference optimization

    Zhanhui Zhou, Jie Liu, Jing Shao, Xiangyu Yue, Chao Yang, Wanli Ouyang, and Yu Qiao. Beyond one-preference-fits-all alignment: Multi-objective direct preference optimization. In Findings of the Association for Computational Linguistics: ACL 2024, pages 10586–10613, 2024

    2024

  37. [37]

    On-the-fly preference alignment via principle-guided decoding

    Mingye Zhu, Yi Liu, Lei Zhang, Junbo Guo, and Zhendong Mao. On-the-fly preference alignment via principle-guided decoding. InThe Thirteenth International Conference on Learning Representations, 2025. Appendix Table of Contents A Detailed Related Work Comparison 13 A.1 Training-based Personalized Alignment . . . . . . . . . . . . . . . . . . . . . . 13 A.2...

    2025

  38. [38]

    the base policyπ base is a relatively poor approximation to the optimal aligned policyπ ∗

  39. [39]

    The bench kid line ride sit

    the merged policy π′ is a better approximation to π∗ than πbase, because it already incorpo- rates the discovered rewards and optimized weights; 3.supp(π ∗)⊆supp(π ′)⊆supp(π base), so the relevant KL divergences are well-defined. Then maximizing the divergence from πbase encourages the updated policy to move away from regions where the base model places p...

  40. [40]

    ( weight : [ w_1 ]) [ P R E F E R E N C E 1 D E S C R I P T I O N ]

  41. [41]

    ( weight : [ w_2 ]) [ P R E F E R E N C E 2 D E S C R I P T I O N ] ... K . ( weight : [ w_K ]) [ P R E F E R E N C E K D E S C R I P T I O N ] User query : [ QUERY ] E.3 Prompt for LLM-as-judge Evaluation For evaluation on the Multifaceted benchmark, we use GPT-4o and claude-sonnet-4-5 as an LLM-as- judge to score each generated response on each assigned...