Pith. sign in

REVIEW 3 major objections 5 minor 62 references

Disentangling Preference Representation and Text Generation for Efficient Individual Preference Alignment

T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Individual LLM preferences can be captured in a small latent vector learned by a tiny encoder, so aligning a model to a new user requires 80–90% less training time than PEFT-based DPO while matching its alignment quality.

desk verdict Genuinely new and practically useful latent-space DPO for efficient personalization, but the latent reward proxy rests on an unstated sampling assumption that needs fixing before publication. read the letter →

arxiv 2412.20834 v1 pith:SBNFQ5YK submitted 2024-12-30 cs.CL cs.AI

Jianfei Zhang , Jun Bai , Bei Li , Yanmeng Wang , Rumei Li , Chenghua Lin , Wenge Rong This is my paper · ORCID
classification cs.CLcs.AI
keywords individualpreferencealignmentlatentDPOvariationalauto-encoderdirectoptimizationparameter-efficientfine-tuningLLMpersonalizationcontrastivelanguage-latentpretrainingtextgeneration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper's claim is that a person's preferences for LLM output can be separated from the machinery of text generation: one low-dimensional latent vector, produced by a small encoder network, can steer a frozen LLM toward what a particular user wants. To get there it first extends a decoder-only LLM into a variational auto-encoder (VAE), training a posterior latent encoder and a latent adapter together with a contrastive objective (CLaP), so that response meaning is captured in a small latent space. Then it applies Direct Preference Optimization (DPO) to that latent space, not to the LLM: preference over responses is converted into preference over latent values through an analytical ratio proxy, and a personalized latent encoder is trained on the resulting latent preference pairs. The authors report that per-user alignment then consumes 80–90% less training time than PEFT-based DPO (LoRA and P-Tuning) while matching or slightly beating its alignment quality on sentiment, dialogue-intention, and summarization tasks. If this holds, serving many individual users becomes a matter of training tiny encoders, with the shared LLM itself never being re-trained for any single person.

What carries the argument

The load-bearing object is the latent reward proxy $\tilde{r}(x,z) = q(z|x,y_w)/q(z|x,y_l)$, where $q(z|x,y)$ is the posterior latent encoder learned in CLaP pretraining. The paper derives it by importance reweighting (Eqs. 7–11): the reward of a latent value is the reward of the responses it would generate, approximated as a weighted average over the preferred and dispreferred responses with weights proportional to $q(z|x,y_k)/p(z|x)$, and because only the comparison matters, the ratio of the two posterior densities suffices. This proxy lets the method rank latent samples by preference without any computation inside the LLM, convert those rankings into latent preference pairs, and run DPO on the latent distribution $p_\theta(z|x)$. The CLaP objective itself — reconstruction plus density-gap KL divergence plus a contrastive term — is what makes the latent space usable; the ablation shows that without the contrastive term or with the standard KL instead of the density-gap version, Latent DPO no longer improves over the base model.

What would settle it

Train Latent DPO on preference pairs deliberately sampled from a distribution far from the model's own response distribution $p(y|x)$ (for example, responses generated by a different model), and compare alignment quality with LoRA-based DPO on the same pairs: if Latent DPO degrades sharply while LoRA-DPO does not, the sampling-distribution assumption is load-bearing. A second check: on held-out preference dimensions, compare the ranking of latent samples by the proxy $\tilde{r}(x,z)=q(z|x,y_w)/q(z|x,y_l)$ against the ranking by the true reward $r(x,y)$; systematic divergence would show the latent reward proxy fails.

Watch

Extended reading notes

Core claim

The central discovery is that preference alignment can be moved out of the LLM entirely. The paper shows a way to infer preferences on latent values from preferences on responses: with one preferred response $y_w$ and one dispreferred response $y_l$, the latent reward $r(x,z)$ can be approximated by an importance-weighted average whose comparison behavior is governed by the posterior ratio $q(z|x,y_w)/q(z|x,y_l)$. That ratio serves as a reward proxy for ranking candidate latent samples, producing latent preference pairs $(x, z_w, z_l)$ on which DPO optimizes the personalized latent encoder $p_\theta(z|x)$ — never the LLM. The authors present experiments on three text-generation tasks with GPT-2, GPT-J-6B, and Llama3-8B backbones, showing that CLaP models match the quality of the original supervised fine-tuned models while Latent DPO matches or exceeds LoRA- and P-Tuning-based DPO in alignment quality at dramatically lower per-user training cost.

Load-bearing premise

The whole pipeline depends on the small latent encoder faithfully capturing what makes one response better than another; if the preference feedback was not drawn from the model's own output distribution, or the encoder misreads a response's meaning, the latent preference pairs it trains on may not reflect the user's real tastes.

Editorial extensions

If this is right

  • Adapting to a new individual user requires training only a small latent encoder (about 0.08–0.29 scaling hours in the paper's measurements, versus 0.47–7.02 for the baselines), so personalization cost scales with the number of users rather than with the size of the LLM.
  • Because the frozen LLM is shared, one latent-adaptive model can serve many users simultaneously, each steered by a different latent vector, and step 2 does not even need access to the LLM's parameters.
  • Alignment quality is maintained or slightly improved relative to LoRA- and P-Tuning-based DPO: on the reported tasks, Latent DPO raised average win-rates or intention probabilities to 83.3, 63.4, and 56.0 from CLaP baselines of 52.5, 25.0, and 46.7, versus LoRA-based DPO reaching 80.8, 62.0, and 55.1.
  • The paper's own stated limitation: the method steers generation within the LLM's existing generative distribution and is not suited to improving foundational model capabilities — it shifts preferences, not capacity.
  • The one-time CLaP pretraining cost (about 3 hours on GPT-2 models and 42 hours on GPT-J-6B in the reported setup) is presented as an investment amortized across users; the per-user win is largest when many individuals share the same base LLM.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable corollary of the disentanglement hypothesis: preferences that the CLaP latent space cannot represent (for example, styles requiring restructuring the base model's knowledge) should resist Latent DPO, which is consistent with the paper's own limitation note.
  • The method should extend to any preference signal expressible as pairwise comparisons, including explicit ratings or multi-turn feedback, since nothing in the derivation requires the reward to come from a human annotator.
  • The 80–90% figure measures the marginal per-user cost with the LLM held fixed; if different users needed different base models or different domains, the upfront CLaP cost would have to be paid again, so the efficiency claim is strongest for many users on one shared model.
  • Because latent preference pairs are built from only two responses per prompt ($K=2$), the quality ceiling may depend on how much preference-relevant signal survives a single preferred/dispreferred comparison; richer preference data could be folded in by increasing $K$ in Eq. 10.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes LatentDPO, a three-stage method for individual preference alignment: CLaP pre-training extends a decoder-only LLM into a conditional VAE with a latent encoder and latent adapter; Latent DPO learns a personalized latent encoder from response-level preference pairs using an approximate latent reward; and the latent adapter steers generation toward personalized outputs. Experiments on IMDB, DailyDialog, and TL;DR with GPT-2 and GPT-J-6B, plus a Llama3-8B study, report 80-90% reductions in per-user adaptation time relative to PEFT baselines and comparable or better alignment quality.

Significance. If the efficiency and quality claims hold, this is a useful contribution because per-user adaptation cost becomes independent of LLM size, which makes large-scale individual alignment more practical. The method is clearly presented, the efficiency numbers are directly measured, and the CLaP ablations in Appendix C and the human evaluation in Section 4.7 are valuable supporting evidence. The main risks are the unverified importance-sampling assumption in Section 3.2.1, a numeric discrepancy in the headline efficiency claim, and the absence of uncertainty quantification on the quality metrics.

major comments (3)
  1. [3.2.1, Eqs. (7)-(11)] The derivation of the latent reward requires the responses used in importance reweighting to be drawn from the base response distribution p(y|x), but the paper never states or verifies this. Section 3.1 samples y_k from pSFT(y|x) for CLaP pre-training, but Algorithm 1 consumes the preference dataset D directly; on TL;DR these pairs come from the Stiennon et al. dataset, whose summaries were generated by a mix of policies, and on IMDB/DailyDialog they are SFT responses filtered by a reward signal. If y_w and y_l are not drawn from p(y|x), Eq. (10) is a biased estimator of r(x,z), and the monotonicity argument leading to r_tilde(x,z) = q(z|x,y_w)/q(z|x,y_l) does not establish that the latent ranking follows the true reward. Please state the sampling assumption explicitly, verify it empirically, or correct the importance weights to account for the actual sampling distribution; otherwise the quality claims in Section 4.5 rest on an unvalidated proxy.
  2. [Abstract, Section 4.5, Fig. 5] The headline claim that the method reduces per-user training time by 80% to 90% is contradicted by the reported DailyDialog numbers: 0.11 hours for Latent DPO versus 0.48 hours for LoRA-based DPO is a 77% relative reduction, outside the claimed range. The range is also wider when P-Tuning is the baseline, with reductions of roughly 86% to 93%. Please revise the claim to match the actual measured range, or explain why a particular baseline or dataset is used for the headline number.
  3. [4.5, Fig. 6 and Table 4] The quality comparisons are reported as single point estimates without error bars, significance tests, or multiple seeds. The TL;DR margin over LoRA-based DPO is 0.9 points (56.0 vs. 55.1), which is small relative to the variability typical of win-rate metrics. Since the central claim includes matching or exceeding PEFT-based alignment quality, please add variance estimates or confidence intervals, and where feasible a paired significance test on the main comparisons, at least for the three primary datasets.
minor comments (5)
  1. [3.2.2, Algorithm 1] The DPO objective is invoked but the reference distribution and the loss scale beta are not specified; please state whether the reference is p(z|x) or the q-initialized encoder, and list the DPO hyperparameters used in the experiments.
  2. [4.7, Table 3] The human evaluation compares Latent DPO with full DPO and SFT, but not with LoRA-based DPO, which is the main comparison in the paper's quality claim; please add a PEFT-based human-evaluation arm or clarify why it is omitted.
  3. [Footnote 3] The footnote says the scores are explained in Section 4.4, but the main results are reported in Section 4.5; the cross-reference should be corrected.
  4. [4.3] The one-time CLaP cost is reported as approximately 3 hours for GPT-2 tasks and 42 hours for TL;DR; please make explicit in the main text that this cost is amortized over all users and is separate from the per-user alignment time reported in Fig. 5.
  5. [Footnote 4] The code URL appears broken across lines in the manuscript ('https://github.com/zhang jf-nlp/LatentDPO'); please ensure the link is complete and accessible.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the latent reward proxy is constructed from independently trained CLaP representations plus external preference labels, and the efficiency and quality claims are measured externally.

full rationale

The paper's central claims are the efficiency of per-user adaptation and the alignment quality of the resulting personalized latent encoder. Neither reduces to its inputs by construction. Step 1 (CLaP) trains the posterior encoder q(z|x,y) and the latent-adaptive generator p(y|x,z) jointly on unlabeled SFT-sampled responses using Eqs. 2-4; the only author-specific component, DG-KLD (Zhang et al., 2022), is a regularization choice whose contribution is separately ablated in Appendix C, and it is not invoked as a uniqueness theorem or to forbid alternatives. Step 2 derives the latent reward via importance sampling in Eqs. 7-11 from Bayes' rule and the variational posterior q; the resulting proxy r_tilde(x,z)=q(z|x,y_w)/q(z|x,y_l) is a function of q and the externally supplied preference pair (x,y_w,y_l), not a fitted parameter renamed as a prediction. Algorithm 1 uses this proxy to compose latent preference pairs and applies standard DPO; initializing p_theta from q is an implementation detail, not a pre-imposition of the answer. Final evaluation is against held-out test splits, off-the-shelf classifiers, GPT-3.5 judgments, and human win-rates (Sections 4.4-4.7), so the quality claim is independently measurable. The reviewer concern that y_w and y_l may not be sampled from p(y|x), as required by the importance-sampling estimator in Eq. 7, is a correctness/robustness assumption about the estimator; it does not make the derivation circular. No load-bearing step equates a fitted input with the predicted quantity, and the authors' self-citation is not used to justify the central claim.

Assumptions & free parameters 6 free parameters · 4 assumptions · 1 invented entities

The method rests on the assumption that preferences are compressible into a 32-dimensional latent space, and that the learned posterior encoder provides reliable importance weights. The latent reward proxy uses a single preference pair, which is a strong approximation. No new physical entities are introduced; the latent variable is a standard VAE construct.

free parameters (6)
  • latent dimension = 32
    Hand-chosen dimensionality of the latent preference space z; not fitted but a key design choice.
  • K for CLaP batch = 4
    Number of responses per prompt used in the reconstruction and contrastive objectives in Eqs. 2 and 3.
  • K for latent reward = 2
    Uses the preferred and dispreferred response pair in Eq. 11, a very coarse importance-sampling estimate.
  • N latent samples = not specified
    Number of prior samples used to construct latent preference pairs in Algorithm 1; the paper does not give the value.
  • posterior encoder transformer layers = 2
    Uses the first 2 transformer layers of the pretrained LLM for q(z|x,y), stated in Section 3.1.
  • CLaP training epochs = 2 (1 frozen, 1 unfrozen)
    Training schedule chosen to balance preserving pretrained ability and learning the latent adapter.
assumptions (4)
  • domain assumption Individual preferences can be faithfully represented by low-dimensional Gaussian latent variables z in R^32
    Stated as a hypothesis in Section 1; the success of the method depends on this capacity.
  • domain assumption The learned posterior q(z|x,y) approximates the true posterior p(z|x,y) well enough for importance weighting
    Invoked in Eq. 8 to replace p(z|x,y) with q(z|x,y) in the importance weights.
  • ad hoc to paper Preference responses y_w and y_l are sampled from the base response distribution p(y|x), making the importance sampling estimator in Eq. 7 valid
    Unstated in Section 3.2.1; if the preference pairs are not drawn from p(y|x), Eq. 11 is biased.
  • domain assumption A 32-dimensional latent condition inserted as KV caches suffices to steer generation without fine-tuning the LLM
    Central architectural assumption of the latent adapter in Section 3.1 and the personalization step.
invented entities (1)
  • Latent preference representation z (32-dimensional)
    purpose: Captures individual user preferences and controls LLM generation through the latent adapter
    Introduced as the medium for preference alignment; no external falsifiable prediction. Internal visualizations (Figs. 7 and 8) show correlation with sentiment and intention, but that is evidence within the paper, not independent.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Disentangling Preference Representation and Text Generation for Efficient Individual Preference Alignment." pith.science (2026). https://pith.science/paper/SBNFQ5YK

@misc{pith2026241220834,
  author       = {Pith},
  title        = {Pith review of: Disentangling Preference Representation and Text Generation for Efficient Individual Preference Alignment},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SBNFQ5YK}},
  note         = {Machine review of arXiv:2412.20834}
}
abstract

Aligning Large Language Models (LLMs) with general human preferences has been proved crucial in improving the interaction quality between LLMs and human. However, human values are inherently diverse among different individuals, making it insufficient to align LLMs solely with general preferences. To address this, personalizing LLMs according to individual feedback emerges as a promising solution. Nonetheless, this approach presents challenges in terms of the efficiency of alignment algorithms. In this work, we introduce a flexible paradigm for individual preference alignment. Our method fundamentally improves efficiency by disentangling preference representation from text generation in LLMs. We validate our approach across multiple text generation tasks and demonstrate that it can produce aligned quality as well as or better than PEFT-based methods, while reducing additional training time for each new individual preference by $80\%$ to $90\%$ in comparison with them.

Figures

Figures reproduced from arXiv: 2412.20834 by the authors.

Figure 1
Figure 1. Our proposed method aims to offer flexible [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Our method realizes efficient personalization for LLMs through three steps. Step 1 learns the posterior [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 4
Figure 4. Illustration of Eq. 3, with condition x omitted. LLM p(y|x, z), and train them on responses sam￾pled from the SFT model through Eq. 4. We name this self-supervised learning procedure as Contrastive Language-Latent Pretraining (CLaP). LCLaP =LReconstruct − LDG-KLD + LContrastive (4) 3.2 Personalization through Latent DPO Given an individual user and their preference data, we aim to learn a personalized latent encoder… view at source ↗
Figures from the paper (5 more)
Figure 5
Figure 5. Figure 5: Additional training time on each new individual preference for different methods. [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Illustration of general performance personal [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Latent representations in CLaP model on IMDB test set. Positive in red and negative in blue [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: Latent representations in the CLaP model on [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]
Figure 9
Figure 9. Figure 9: The Pearson correlation coefficient between different preferences on summarization. [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

62 extracted references · 24 canonical work pages

  1. [1]

    Brown, Jack Clark, Sam McCandlish, Chris Olah, Benjamin Mann, and Jared Kaplan

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El Showk, Nelson Elhage, Zac Hatfield - Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson...

    arXiv doi:10.48550/arxiv.2204.05862 2022

  2. [2]

    Siqi Bao, Huang He, Fan Wang, Hua Wu, and Haifeng Wang. 2020. https://doi.org/10.18653/v1/2020.acl-main.9 PLATO: pre-trained dialogue generation model with discrete latent variable . In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020 , pages 85--96

    doi:10.18653/v1/2020.acl-main.9 2020

  3. [3]

    Souradip Chakraborty, Jiahao Qiu, Hui Yuan, Alec Koppel, Furong Huang, Dinesh Manocha, Amrit Singh Bedi, and Mengdi Wang. 2024. https://doi.org/10.48550/arXiv.2402.08925 Maxmin-rlhf: Towards equitable alignment of large language models with diverse human preferences . CoRR, abs/2402.08925

    arXiv doi:10.48550/arxiv.2402.08925 2024

  4. [4]

    Wei Chen, Yeyun Gong, Song Wang, Bolun Yao, Weizhen Qi, Zhongyu Wei, Xiaowu Hu, Bartuer Zhou, Yi Mao, Weizhu Chen, Biao Cheng, and Nan Duan. 2022. https://doi.org/10.18653/v1/2022.acl-long.333 Dialogved: A pre-trained latent variable encoder-decoder model for dialog response generation . In Proceedings of the 60th Annual Meeting of the Association for Com...

    doi:10.18653/v1/2022.acl-long.333 2022

  5. [5]

    Pengyu Cheng, Jiawen Xie, Ke Bai, Yong Dai, and Nan Du. 2023. https://doi.org/10.48550/arXiv.2309.03126 Everyone deserves A reward: Learning customized human preferences . CoRR, abs/2309.03126

    arXiv doi:10.48550/arxiv.2309.03126 2023

  6. [6]

    Christiano, Jan Leike, Tom B

    Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. https://proceedings.neurips.cc/paper/2017/hash/d5e2c0adad503c91f91df240d0cd4e49-Abstract.html Deep reinforcement learning from human preferences . In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing System...

    2017

  7. [7]

    Hanze Dong, Wei Xiong, Deepanshu Goyal, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. 2023. https://openreview.net/forum?id=m7p5O7zblY RAFT: reward ranked finetuning for generative foundation model alignment . CoRR, abs/2304.06767

    arXiv 2023

  8. [8]

    Yu Duan, Canwen Xu, Jiaxin Pei, Jialong Han, and Chenliang Li. 2020. https://doi.org/10.18653/v1/2020.acl-main.23 Pre-train and plug-in: Flexible conditional text generation with variational auto-encoders . In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020 , pages 253--262

    doi:10.18653/v1/2020.acl-main.23 2020

Show all 62 references
  1. [9]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al - Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Z...

    2024 doi arXiv

  2. [10]

    Le Fang, Tao Zeng, Chaochun Liu, Liefeng Bo, Wen Dong, and Changyou Chen. 2021. https://arxiv.org/abs/2101.00828 Transformer-based conditional variational autoencoder for controllable story generation . CoRR, abs/2101.00828

    2021 arXiv

  3. [11]

    Alexander Havrilla, Maksym Zhuravinskyi, Duy Phung, Aman Tiwari, Jonathan Tow, Stella Biderman, Quentin Anthony, and Louis Castricato. 2023. https://aclanthology.org/2023.emnlp-main.530 trl X : A framework for large scale reinforcement learning from human feedback . In Proceed...

    2023

  4. [12]

    Matthew D Hoffman and Matthew J Johnson. 2016. Elbo surgery: yet another way to carve up the variational evidence lower bound. In Workshop in Advances in Approximate Bayesian Inference, NIPS

    2016

  5. [13]

    Tom Hosking, Phil Blunsom, and Max Bartolo. 2024. https://openreview.net/forum?id=7W3GLNImfS Human feedback is not gold standard . In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024

    2024

  6. [14]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen - Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen - Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022 a . https://openreview.net/forum?id=nZeVKeeFYf9 Lora: Low-rank adaptation of large language models . In The Tenth International Conference on Learning Represen...

    2022

  7. [15]

    Jinyi Hu, Xiaoyuan Yi, Wenhao Li, Maosong Sun, and Xing Xie. 2022 b . https://doi.org/10.18653/v1/2022.naacl-main.51 Fuse it more deeply! A variational transformer with layer-wise latent variable inference for text generation . In Proceedings of the 2022 Conference of the Nort...

    2022 doi

  8. [16]

    Zhiting Hu, Zichao Yang, Xiaodan Liang, Ruslan Salakhutdinov, and Eric P. Xing. 2017. http://proceedings.mlr.press/v70/hu17e.html Toward controlled generation of text . In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, ...

    2017

  9. [17]

    Joel Jang, Seungone Kim, Bill Yuchen Lin, Yizhong Wang, Jack Hessel, Luke Zettlemoyer, Hannaneh Hajishirzi, Yejin Choi, and Prithviraj Ammanabrolu. 2023. Personalized soups: Personalized large language model alignment via post-hoc parameter merging. CoRR, abs/2310.11564

    2023 arXiv

  10. [18]

    Vineet John, Lili Mou, Hareesh Bahuleyan, and Olga Vechtomova. 2019. https://doi.org/10.18653/v1/p19-1041 Disentangled representation learning for non-parallel text style transfer . In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 201...

    2019 doi

  11. [19]

    Seungone Kim, Jamin Shin, Yejin Choi, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thorne, and Minjoon Seo. 2024 a . https://openreview.net/forum?id=8euJaTveKw Prometheus: Inducing fine-grained evaluation capability in language models ...

    2024

  12. [20]

    Seungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin, Jamin Shin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, and Minjoon Seo. 2024 b . https://doi.org/10.48550/arXiv.2405.01535 Prometheus 2: An open source language model specialized in evaluating other langu...

    2024 doi arXiv

  13. [21]

    Kingma and Max Welling

    Diederik P. Kingma and Max Welling. 2014. http://arxiv.org/abs/1312.6114 Auto-encoding variational bayes . In 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings

    2014 arXiv

  14. [22]

    Seongyun Lee, Sue Hyun Park, Seungone Kim, and Minjoon Seo. 2024. https://doi.org/10.48550/arXiv.2405.17977 Aligning to thousands of preferences via system message generalization . CoRR, abs/2405.17977

    2024 doi arXiv

  15. [23]

    Chunyuan Li, Xiang Gao, Yuan Li, Baolin Peng, Xiujun Li, Yizhe Zhang, and Jianfeng Gao. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.378 Optimus: Organizing sentences via pre-trained modeling of a latent space . In Proceedings of the 2020 Conference on Empirical Methods i...

    2020 doi

  16. [24]

    Dengchun Li, Yingzi Ma, Naizheng Wang, Zhiyuan Cheng, Lei Duan, Jie Zuo, Cal Yang, and Mingjie Tang. 2024 a . Mixlora: Enhancing large language models fine-tuning with lora based mixture of experts. CoRR, abs/2404.15159

    2024 arXiv

  17. [25]

    Dexun Li, Cong Zhang, Kuicai Dong, Derrick - Goh - Xin Deik, Ruiming Tang, and Yong Liu. 2024 b . Aligning crowd feedback via distributional preference reward modeling. CoRR, abs/2402.09764

    2024 arXiv

  18. [26]

    Ming Li, Jiuhai Chen, Lichang Chen, and Tianyi Zhou. 2024 c . https://aclanthology.org/2024.findings-acl.956 Can llms speak for diverse people? tuning llms via debate to generate controllable controversial statements . CoRR, abs/2402.10614

    2024 arXiv

  19. [27]

    Xiang Lisa Li and Percy Liang. 2021. https://doi.org/10.18653/v1/2021.acl-long.353 Prefix-tuning: Optimizing continuous prompts for generation . In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conferen...

    2021 doi

  20. [28]

    Lipton, and Liu Leqi

    Xinyu Li, Zachary C. Lipton, and Liu Leqi. 2024 d . Personalized language modeling from personalized human feedback. CoRR, abs/2402.05133

    2024 arXiv

  21. [29]

    Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. 2017. https://aclanthology.org/I17-1099/ Dailydialog: A manually labelled multi-turn dialogue dataset . In Proceedings of the Eighth International Joint Conference on Natural Language Processing, IJCNLP 201...

    2017

  22. [30]

    Hao Liu, Carmelo Sferrazza, and Pieter Abbeel. 2024. https://openreview.net/forum?id=6xfe4IVcOu Chain of hindsight aligns language models with feedback . In The Twelfth International Conference on Learning Representations

    2024

  23. [31]

    Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. 2022. https://doi.org/10.18653/v1/2022.acl-short.8 P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks . In Proceedings of the 60th Annual Meeting of the Associati...

    2022 doi

  24. [32]

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. http://arxiv.org/abs/1907.11692 Roberta: A robustly optimized BERT pretraining approach . CoRR, abs/1907.11692

    2019 arXiv

  25. [33]

    Maas, Raymond E

    Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. https://aclanthology.org/P11-1015/ Learning word vectors for sentiment analysis . In The 49th Annual Meeting of the Association for Computational Linguistics: Human Language T...

    2011

  26. [34]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leik...

    2022

  27. [35]

    Seongmin Park and Jihwa Lee. 2021. https://arxiv.org/abs/2108.02446 Finetuning pretrained transformers into variational autoencoders . CoRR, abs/2108.02446

    2021 arXiv

  28. [36]

    Fernando A Quintana, Jun S Liu, and Guido E del Pino. 1999. https://www.sciencedirect.com/science/article/pii/S0167947398000759 Monte carlo em with importance reweighting and its applications in random effects models . Computational statistics & data analysis, 29(4):429--444

    1999

  29. [37]

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9

    2019

  30. [38]

    Manning, Stefano Ermon, and Chelsea Finn

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn. 2023. http://papers.nips.cc/paper\_files/paper/2023/hash/a85b405ed65c6477a4fe8302b5e06ce7-Abstract-Conference.html Direct preference optimization: Your language model is sec...

    2023

  31. [39]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. http://jmlr.org/papers/v21/20-074.html Exploring the limits of transfer learning with a unified text-to-text transformer . J. Mach. Learn. Res., ...

    2020

  32. [40]

    Rajkumar Ramamurthy, Prithviraj Ammanabrolu, Kiant \' e Brantley, Jack Hessel, Rafet Sifa, Christian Bauckhage, Hannaneh Hajishirzi, and Yejin Choi. 2023. https://openreview.net/forum?id=8aHzds2uUyB Is reinforcement learning (not) for natural language processing: Benchmarks, b...

    2023

  33. [41]

    Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. 2014. http://proceedings.mlr.press/v32/rezende14.html Stochastic backpropagation and approximate inference in deep generative models . In Proceedings of the 31th International Conference on Machine Learning, ICML 2014,...

    2014

  34. [42]

    Pranab Sahoo, Ayush Kumar Singh, Sriparna Saha, Vinija Jain, Samrat Mondal, and Aman Chadha. 2024. https://doi.org/10.48550/arXiv.2402.07927 A systematic survey of prompt engineering in large language models: Techniques and applications . CoRR, abs/2402.07927

    2024 doi arXiv

  35. [43]

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. http://arxiv.org/abs/1707.06347 Proximal policy optimization algorithms . CoRR, abs/1707.06347

    2017 arXiv

  36. [44]

    Joar Skalse, Nikolaus H. R. Howe, Dmitrii Krasheninnikov, and David Krueger. 2022. https://doi.org/10.48550/arXiv.2209.13085 Defining and characterizing reward hacking . CoRR, abs/2209.13085

    2022 doi arXiv

  37. [45]

    Kihyuk Sohn, Honglak Lee, and Xinchen Yan. 2015. https://proceedings.neurips.cc/paper/2015/hash/8d55a249e6baa5c06772297520da2051-Abstract.html Learning structured output representation using deep conditional generative models . In Advances in Neural Information Processing Syst...

    2015

  38. [46]

    Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F

    Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F. Christiano. 2020. https://arxiv.org/abs/2009.01325 Learning to summarize from human feedback . CoRR, abs/2009.01325

    2020 arXiv

  39. [47]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton - Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu,...

    2023 doi arXiv

  40. [48]

    A \" a ron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. http://arxiv.org/abs/1807.03748 Representation learning with contrastive predictive coding . CoRR, abs/1807.03748

    2018 arXiv

  41. [49]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. https://proceedings.neurips.cc/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html Attention is all you need . In Advances in Neural I...

    2017

  42. [50]

    Chaoqi Wang, Yibo Jiang, Chenghao Yang, Han Liu, and Yuxin Chen. 2024. https://openreview.net/forum?id=2cRzmWXK9N Beyond reverse KL: generalizing direct preference optimization with diverse divergence constraints . In The Twelfth International Conference on Learning Representa...

    2024

  43. [51]

    Guan Wang, Sijie Cheng, Xianyuan Zhan, Xiangang Li, Sen Song, and Yang Liu. 2023. https://openreview.net/forum?id=AOJyfhWYHf Openchat: Advancing open-source language models with mixed-quality data . CoRR, abs/2309.11235

    2023 arXiv

  44. [52]

    Smith, Mari Ostendorf, and Hannaneh Hajishirzi

    Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A. Smith, Mari Ostendorf, and Hannaneh Hajishirzi. 2023. Fine-grained human feedback gives better rewards for language model training. In Advances in Neural Information Processing Systems 36:...

    2023

  45. [53]

    Peng Xu, Jackie Chi Kit Cheung, and Yanshuai Cao. 2020. http://proceedings.mlr.press/v119/xu20a.html On variational learning of controllable representations for text without supervision . In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18...

    2020

  46. [54]

    Seonghyeon Ye, Doyoung Kim, Sungdong Kim, Hyeonbin Hwang, Seungone Kim, Yongrae Jo, James Thorne, Juho Kim, and Minjoon Seo. 2024. https://openreview.net/forum?id=CYmF38ysDa FLASK: fine-grained language model evaluation based on alignment skill sets . In The Twelfth Internatio...

    2024

  47. [55]

    Xiaoyuan Yi, Ruoyu Li, Cheng Yang, Wenhao Li, and Maosong Sun. 2020. https://doi.org/10.1609/aaai.v34i05.6488 Mixpoet: Diverse poetry generation via learning controllable mixed latent space . In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirt...

    2020 doi

  48. [56]

    Meng - Hsuan Yu, Juntao Li, Danyang Liu, Bo Tang, Haisong Zhang, Dongyan Zhao, and Rui Yan. 2020. https://doi.org/10.1609/aaai.v34i02.5538 Draft and edit: Automatic storytelling through multi-pass hierarchical conditional variational autoencoder . In The Thirty-Fourth AAAI Con...

    2020 doi

  49. [57]

    Jianfei Zhang, Jun Bai, Chenghua Lin, Yanmeng Wang, and Wenge Rong. 2022. http://papers.nips.cc/paper\_files/paper/2022/hash/7b2e844c52349134268e819a9b56b9e8-Abstract-Conference.html Improving variational autoencoders with density gap-based regularization . In Advances in Neur...

    2022

  50. [58]

    Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. 2023. https://openreview.net/forum?id=lq62uWRJjiY Adaptive budget allocation for parameter-efficient fine-tuning . In The Eleventh International Conference on Learning Representa...

    2023

  51. [59]

    Tiancheng Zhao, Ran Zhao, and Maxine Esk \' e nazi. 2017. https://doi.org/10.18653/v1/P17-1061 Learning discourse-level diversity for neural dialog models using conditional variational autoencoders . In Proceedings of the 55th Annual Meeting of the Association for Computationa...

    2017 doi

  52. [60]

    Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B

    Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul F. Christiano, and Geoffrey Irving. 2019. http://arxiv.org/abs/1909.08593 Fine-tuning language models from human preferences . CoRR, abs/1909.08593

    2019 arXiv

  53. [61]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  54. [62]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.