pith. sign in

arxiv: 2605.23244 · v1 · pith:JQ7B4CGAnew · submitted 2026-05-22 · 💻 cs.LG

Convex Optimization for Alignment and Preference Learning on a Single GPU

Miria Feng , Mert Pilanci This is my paper

Pith reviewed 2026-05-25 04:59 UTC · model grok-4.3

classification 💻 cs.LG
keywords convex optimizationpreference optimizationLLM alignmentsingle GPUDPOORPOneural network reformulationreward stability
0
0 comments X

The pith

COALA applies convex optimization to preference fine-tuning of LLMs to remove the reference model and enable single-GPU training.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces COALA, a method that reformulates the alignment of large language models with human preferences as a convex optimization problem. This reformulation eliminates the reference model required by approaches like DPO and reduces both training time and memory usage enough to run on a single GPU. Experiments on four datasets and six models, including Llama-3.1-8B, show competitive results with far less compute—about 17.6% of DPO's TFLOPs—and more stable reward curves that peak faster. The work claims this is the first effective use of convex optimization for LLM preference fine-tuning.

Core claim

By leveraging the convex optimization reformulation of neural networks, COALA solves the preference fine-tuning objective without a reference model, achieving strong theoretical guarantees, significant reductions in training time and VRAM consumption, and stable monotonically increasing rewards that reach peak margins quicker than DPO or ORPO, all while maintaining competitive performance across multiple datasets and models.

What carries the argument

The convex optimization reformulation of neural networks applied to the preference fine-tuning objective, which removes the reference model and enables efficient single-GPU computation.

Load-bearing premise

The convex optimization reformulation of neural networks can be applied directly to preference fine-tuning while preserving theoretical guarantees and empirical performance.

What would settle it

A run on the Llama-3.1-8B model with the Educational Feedback dataset where COALA requires multiple GPUs or shows lower reward margins than DPO.

Figures

Figures reproduced from arXiv: 2605.23244 by Mert Pilanci, Miria Feng.

Figure 1
Figure 1. Figure 1: COALA shows stable reward margin gains across all models. This is attributed to its theoretically grounded foundation which alleviates fine-tuning reliance on hyperparameter tuning and general heuristics, as well as stability derived through the convex program. METHOD LC WR % WR % AVG LENGTH EDU IMDB ULTRA EDU IMDB ULTRA EDU IMDB ULTRA Mistral-7B Model COALA 24.61±0.30 24.88±1.46 20.84±1.35 23.82±1.38 23.1… view at source ↗
Figure 2
Figure 2. Figure 2: Alternating Population Method for creating preference datasets. One conversation yields multiple (prompt, chosen, re￾jected) preference triplets, without requiring external LLMs to generate matching responses in chosen-rejected pairs. 6.3. Alternating Population Method for Datasets We exploit the structure of multi-turn conversational layouts to design a sample-efficient preference extraction method via a … view at source ↗
Figure 3
Figure 3. Figure 3: COALA and DPO mean reward margins for runs on the IMDb Dataset. The DPO Train variant displays Time Weighted Exponential Moving Average (EMA) smoothing. 17 [PITH_FULL_IMAGE:figures/full_fig_p017_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: COALA and DPO mean reward margins for runs on the EduFeedback Dataset. The DPO Train variant displays Time Weighted Exponential Moving Average (EMA) smoothing [PITH_FULL_IMAGE:figures/full_fig_p018_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Scatter plot of LC WR% versus TFLOPs. COALA uses less TFLOPs to achieve high LC WR% across all three models. 18 [PITH_FULL_IMAGE:figures/full_fig_p018_5.png] view at source ↗
read the original abstract

Fine-tuning large language models (LLMs) to align with human preferences has driven the success of systems such as Gemini and ChatGPT. However, approaches like Reinforcement Learning from Human Feedback (RLHF) remain computationally expensive and complex. Direct Preference Optimization (DPO) offers a simpler alternative but has limitations such as inconsistent ranking accuracy, high dependence on GPU resources, and expensive hyperparameter tuning. We propose the Convex Optimization for Alignment and Preference Learning Algorithm (COALA): a novel lightweight strategy with strong theoretical guarantees. By leveraging the convex optimization reformulation of neural networks, COALA eliminates the need for a reference model and obtains significant reduction in both training time and VRAM consumption, thus enabling efficient training on a single GPU. Experiments across four datasets--including a 26621-sample synthetic Educational Feedback dataset--and six models (including Llama-3.1-8B) demonstrate COALA's competitive performance and efficiency while utilizing as little as ~17.6% of DPO's total TFLOPs. COALA exhibits stable, monotonically increasing rewards and reaches peak margins in significantly shorter time in comparison to traditional methods such as DPO and ORPO. To the best of our knowledge, this is the first time convex optimization has been effectively applied to preference fine-tuning of LLMs.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The paper introduces COALA, a lightweight preference fine-tuning method for LLMs that applies a convex optimization reformulation of neural networks to the alignment objective. It claims this eliminates the reference model, yields strong theoretical guarantees, reduces training time and VRAM to enable single-GPU training (as low as ~17.6% of DPO's TFLOPs), produces stable monotonically increasing rewards that reach peak margins faster than DPO and ORPO, and achieves competitive performance across four datasets (including a 26k-sample synthetic Educational Feedback set) and six models up to Llama-3.1-8B.

Significance. If the convex reformulation can be substituted into the preference objective while preserving expressivity, ranking properties, and the claimed guarantees, the result would meaningfully lower the computational barrier for alignment research. The reported single-GPU feasibility, TFLOPs reduction, and multi-model/multi-dataset empirical evaluation would be notable strengths if the experimental controls and equivalence arguments are rigorous.

major comments (2)
  1. [Abstract and §3] Abstract and §3 (method): the central claim that the convex NN reformulation can be directly substituted into the Bradley-Terry-style preference loss while retaining both non-convex model expressivity and 'strong theoretical guarantees' is load-bearing but unsupported by any derivation, equivalence proof, or approximation bound in the provided text; without this the monotonic reward and efficiency claims rest on an unverified assumption.
  2. [§4] §4 (experiments): the efficiency comparison (17.6% TFLOPs vs DPO) and single-GPU claim require explicit controls for batch size, sequence length, optimizer settings, and whether the convex surrogate alters the implicit reward model; absent these, the quantitative gains cannot be isolated from implementation differences.
minor comments (2)
  1. [Abstract] The abstract states 'to the best of our knowledge, this is the first time convex optimization has been effectively applied to preference fine-tuning' but does not cite or distinguish from prior convex reformulations of NNs that the method builds upon.
  2. [§3] Notation for the convex reformulation (e.g., how the original non-convex parameters map to the convex surrogate) is not introduced with sufficient clarity for reproducibility.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the detailed and constructive feedback. We address the two major comments point by point below and will revise the manuscript to strengthen the theoretical justification and experimental documentation.

read point-by-point responses

  1. Referee: [Abstract and §3] Abstract and §3 (method): the central claim that the convex NN reformulation can be directly substituted into the Bradley-Terry-style preference loss while retaining both non-convex model expressivity and 'strong theoretical guarantees' is load-bearing but unsupported by any derivation, equivalence proof, or approximation bound in the provided text; without this the monotonic reward and efficiency claims rest on an unverified assumption.

    Authors: Section 3 presents the convex reformulation and its direct substitution into the preference objective, arguing that the reformulation preserves the original network's ranking properties while converting the optimization to a convex problem. We acknowledge, however, that an explicit derivation, equivalence statement, or approximation bound is not provided. We will add a dedicated subsection in the revised manuscript that derives the substitution, shows retention of expressivity for ranking, and states the resulting theoretical guarantees. revision: yes

  2. Referee: [§4] §4 (experiments): the efficiency comparison (17.6% TFLOPs vs DPO) and single-GPU claim require explicit controls for batch size, sequence length, optimizer settings, and whether the convex surrogate alters the implicit reward model; absent these, the quantitative gains cannot be isolated from implementation differences.

    Authors: The reported TFLOPs and single-GPU results were obtained under matched conditions (identical batch size, sequence length, and optimizer settings) across methods, with the convex surrogate applied only to the loss while leaving the underlying reward model unchanged. We agree that these controls should be stated more explicitly. We will expand Section 4 with a table or paragraph that lists the exact hyper-parameters, confirms the reward-model equivalence, and isolates the contribution of the reformulation. revision: yes

Circularity Check

0 steps flagged

No significant circularity detected

full rationale

The paper applies an existing convex optimization reformulation of neural networks to the preference fine-tuning objective, claiming strong theoretical guarantees and efficiency gains. No equations, derivations, or steps in the provided abstract or description reduce any prediction, guarantee, or central result to a fitted input, self-definition, or self-citation chain by construction. The reformulation is leveraged as an established technique applied to a new domain (LLM alignment), with empirical validation across datasets and models serving as independent support. This is a standard non-circular application of prior methods, warranting a score of 0.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central claim depends on the applicability of prior convex neural-network reformulations to preference objectives; no new free parameters or invented entities are explicitly introduced in the abstract.

axioms (1)
  • domain assumption Convex optimization reformulation of neural networks applies to the preference fine-tuning objective
    Invoked as the foundation of COALA in the abstract.

pith-pipeline@v0.9.0 · 5755 in / 1267 out tokens · 22222 ms · 2026-05-25T04:59:28.812779+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

135 extracted references · 40 canonical work pages · 20 internal anchors

  1. [1]

    Know What You Don ' t Know: Unanswerable Questions for SQ u AD

    Rajpurkar, Pranav and Jia, Robin and Liang, Percy. Know What You Don ' t Know: Unanswerable Questions for SQ u AD. Association for Computational Linguistics (ACL). 2018

    2018

  2. [2]

    Advances in neural information processing systems , volume=

    Language models are few-shot learners , author=. Advances in neural information processing systems , volume=

  3. [3]

    Advances in neural information processing systems , volume=

    Judging llm-as-a-judge with mt-bench and chatbot arena , author=. Advances in neural information processing systems , volume=

  4. [4]

    arXiv preprint arXiv:2511.10985 , year=

    When Data is the Algorithm: A Systematic Study and Curation of Preference Optimization Datasets , author=. arXiv preprint arXiv:2511.10985 , year=

    work page arXiv

  5. [5]

    The Llama 3 Herd of Models

    The llama 3 herd of models , author=. arXiv preprint arXiv:2407.21783 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  6. [6]

    2023 , eprint=

    SteerLM: Attribute Conditioned SFT as an (User-Steerable) Alternative to RLHF , author=. 2023 , eprint=

    2023

  7. [7]

    DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models , author=. arXiv preprint arXiv:2402.03300 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  8. [8]

    Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) , pages=

    Helpsteer: Multi-attribute helpfulness dataset for steerlm , author=. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) , pages=

    2024

  9. [9]

    The Curious Case of Neural Text Degeneration

    The curious case of neural text degeneration , author=. arXiv preprint arXiv:1904.09751 , year=

    work page internal anchor Pith review Pith/arXiv arXiv 1904

  10. [10]

    Advances in Neural Information Processing Systems , volume=

    Globally optimal training of generalized polynomial neural networks with nonlinear spectral methods , author=. Advances in Neural Information Processing Systems , volume=

  11. [11]

    arXiv preprint arXiv:2409.01281 , year=

    Path-consistency with prefix enhancement for efficient inference in llms , author=. arXiv preprint arXiv:2409.01281 , year=

    work page arXiv

  12. [12]

    International Conference on Machine Learning , pages=

    Decoding-time Realignment of Language Models , author=. International Conference on Machine Learning , pages=. 2024 , organization=

    2024

  13. [13]

    arXiv preprint arXiv:2509.26169 , year=

    Alignment-Aware Decoding , author=. arXiv preprint arXiv:2509.26169 , year=

    work page arXiv

  14. [14]

    OpenAI blog , volume=

    Language models are unsupervised multitask learners , author=. OpenAI blog , volume=

  15. [15]

    the method of paired comparisons , author=

    Rank analysis of incomplete block designs: I. the method of paired comparisons , author=. Biometrika , volume=. 1952 , publisher=

    1952

  16. [16]

    Advances in Neural Information Processing Systems , volume=

    Preference learning algorithms do not learn preference rankings , author=. Advances in Neural Information Processing Systems , volume=

  17. [17]

    Advances in Neural Information Processing Systems , volume=

    Simpo: Simple preference optimization with a reference-free reward , author=. Advances in Neural Information Processing Systems , volume=

  18. [18]

    Advances in neural information processing systems , volume=

    Training language models to follow instructions with human feedback , author=. Advances in neural information processing systems , volume=

  19. [19]

    Advances in neural information processing systems , volume=

    Deep reinforcement learning from human preferences , author=. Advances in neural information processing systems , volume=

  20. [20]

    Advances in neural information processing systems , volume=

    Learning to summarize with human feedback , author=. Advances in neural information processing systems , volume=

  21. [21]

    Advances in neural information processing systems , volume=

    Pytorch: An imperative style, high-performance deep learning library , author=. Advances in neural information processing systems , volume=

  22. [22]

    Equinox: neural networks in JAX via callable PyTrees and filtered transformations

    Equinox: neural networks in JAX via callable PyTrees and filtered transformations , author=. arXiv preprint arXiv:2111.00254 , year=

    work page arXiv

  23. [23]

    IFAC Proceedings Volumes , volume=

    An ADMM algorithm for a class of total variation regularized estimation problems , author=. IFAC Proceedings Volumes , volume=. 2012 , publisher=

    2012

  24. [24]

    Computational optimization and applications , volume=

    Generalized symmetric ADMM for separable convex optimization , author=. Computational optimization and applications , volume=. 2018 , publisher=

    2018

  25. [25]

    GPT-4 Technical Report

    Gpt-4 technical report , author=. arXiv preprint arXiv:2303.08774 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  26. [26]

    Adam: A Method for Stochastic Optimization

    Adam: A method for stochastic optimization , author=. arXiv preprint arXiv:1412.6980 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  27. [27]

    arXiv preprint arXiv:2006.14799 , year=

    Evaluation of text generation: A survey , author=. arXiv preprint arXiv:2006.14799 , year=

    work page arXiv 2006

  28. [28]

    International conference on learning representation , year=

    Rmsprop converges with proper hyperparameter , author=. International conference on learning representation , year=

  29. [29]

    International conference on machine learning , pages=

    Do imagenet classifiers generalize to imagenet? , author=. International conference on machine learning , pages=. 2019 , organization=

    2019

  30. [30]

    ultralytics/yolov5: v4. 0-nn. SiLU () activations, Weights & Biases logging, PyTorch Hub integration , author=. Zenodo , year=

  31. [31]

    Understanding Dataset Difficulty with

    Ethayarajh, Kawin and Choi, Yejin and Swayamdipta, Swabha , booktitle =. Understanding Dataset Difficulty with. 2022 , volume =

    2022

  32. [32]

    Advances in Neural Information Processing Systems , volume=

    Lima: Less is more for alignment , author=. Advances in Neural Information Processing Systems , volume=

  33. [33]

    2020 12th International Conference on Computational Intelligence and Communication Networks (CICN) , pages=

    Analyzing sentiment using IMDb dataset , author=. 2020 12th International Conference on Computational Intelligence and Communication Networks (CICN) , pages=. 2020 , organization=

    2020

  34. [34]

    Journal of Global Optimization , volume=

    Generalized risk parity portfolio optimization: An ADMM approach , author=. Journal of Global Optimization , volume=. 2020 , publisher=

    2020

  35. [35]

    MNIST-C: A Robustness Benchmark for Computer Vision

    Mnist-c: A robustness benchmark for computer vision , author=. arXiv preprint arXiv:1906.02337 , year=

    work page internal anchor Pith review Pith/arXiv arXiv 1906

  36. [36]

    Signal Processing: Image Communication , volume=

    ADMM for image restoration based on nonlocal simultaneous sparse Bayesian coding , author=. Signal Processing: Image Communication , volume=. 2019 , publisher=

    2019

  37. [37]

    Convergence Analysis and Design of Multi-block ADMM via Switched Control Theory

    Convergence Analysis and Design of multi-block ADMM via switched control theory , author=. arXiv preprint arXiv:1709.05528 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  38. [38]

    Do CIFAR-10 Classifiers Generalize to CIFAR-10?

    Do cifar-10 classifiers generalize to cifar-10? , author=. arXiv preprint arXiv:1806.00451 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  39. [39]

    RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback

    Rlaif: Scaling reinforcement learning from human feedback with ai feedback , author=. arXiv preprint arXiv:2309.00267 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  40. [40]

    PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel

    Pytorch fsdp: experiences on scaling fully sharded data parallel , author=. arXiv preprint arXiv:2304.11277 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  41. [41]

    Advances in Neural Information Processing Systems , volume=

    Jax md: a framework for differentiable physics , author=. Advances in Neural Information Processing Systems , volume=

  42. [42]

    Advances in neural information processing systems , volume=

    Convex neural networks , author=. Advances in neural information processing systems , volume=

  43. [43]

    , Title =

    OpenAI, Inc. , Title =

  44. [44]

    arXiv preprint arXiv:2110.08460 , year=

    A short study on compressing decoder-based language models , author=. arXiv preprint arXiv:2110.08460 , year=

    work page arXiv

  45. [45]

    Advances in Neural Information Processing Systems , volume=

    Direct preference optimization: Your language model is secretly a reward model , author=. Advances in Neural Information Processing Systems , volume=

  46. [46]

    arXiv preprint arXiv:2306.14111 , year=

    Is RLHF More Difficult than Standard RL? , author=. arXiv preprint arXiv:2306.14111 , year=

    work page arXiv

  47. [47]

    Iraqi Journal For Computer Science and Mathematics , volume=

    ChatGpt: open possibilities , author=. Iraqi Journal For Computer Science and Mathematics , volume=

  48. [48]

    Comparison Between RLHF and RLAIF in Fine-Tuning a Large Language Model , author=

  49. [49]

    Joint European Conference on Machine Learning and Knowledge Discovery in Databases , pages=

    Marl-jax: Multi-agent reinforcement leaning framework for social generalization , author=. Joint European Conference on Machine Learning and Knowledge Discovery in Databases , pages=. 2023 , organization=

    2023

  50. [50]

    DPO: Direct Preference Optimization , booktitle=

  51. [51]

    World Patent Information , volume=

    Patent claim generation by fine-tuning OpenAI GPT-2 , author=. World Patent Information , volume=. 2020 , publisher=

    2020

  52. [52]

    Weights and Biases , booktitle=

  53. [53]

    Foundations and Trends

    Distributed optimization and statistical learning via the alternating direction method of multipliers , author=. Foundations and Trends. 2011 , publisher=

    2011

  54. [54]

    Hugging Face , booktitle=

  55. [55]

    International Conference on Machine Learning , pages=

    Neural networks are convex regularizers: Exact polynomial-time convex optimization formulations for two-layer networks , author=. International Conference on Machine Learning , pages=. 2020 , organization=

    2020

  56. [56]

    GPT4: The ultimate brain , author=

  57. [57]

    Advances in neural information processing systems , volume=

    Attention is all you need , author=. Advances in neural information processing systems , volume=

  58. [58]

    Mathematical Programming , volume=

    Lower bounds for non-convex stochastic optimization , author=. Mathematical Programming , volume=. 2023 , publisher=

    2023

  59. [59]

    Numerische Mathematik , volume=

    On non-ergodic convergence rate of Douglas--Rachford alternating direction method of multipliers , author=. Numerische Mathematik , volume=. 2015 , publisher=

    2015

  60. [60]

    SIAM Journal on Numerical Analysis , volume=

    On the O(1/n) convergence rate of the Douglas--Rachford alternating direction method , author=. SIAM Journal on Numerical Analysis , volume=. 2012 , publisher=

    2012

  61. [61]

    SIAM Journal on Mathematics of Data Science , volume=

    Efficient global optimization of two-layer relu networks: Quadratic-time algorithms and adversarial training , author=. SIAM Journal on Mathematics of Data Science , volume=. 2023 , publisher=

    2023

  62. [62]

    International Conference on Machine Learning , pages=

    Fast convex optimization for two-layer relu networks: Equivalent model classes and cone decompositions , author=. International Conference on Machine Learning , pages=. 2022 , organization=

    2022

  63. [63]

    International Conference on Machine Learning , year =

    Convex Relaxations of ReLU Neural Networks Approximate Global Optima in Polynomial Time , author=. International Conference on Machine Learning , year =

  64. [64]

    Journal of Parallel and Distributed Computing , volume=

    GPU acceleration of ADMM for large-scale quadratic programming , author=. Journal of Parallel and Distributed Computing , volume=. 2020 , publisher=

    2020

  65. [65]

    Genios: an (almost) second-order operator-splitting solver for large-scale convex optimization

    GeNIOS: an (almost) second-order operator-splitting solver for large-scale convex optimization , author=. arXiv preprint arXiv:2310.08333 , year=

    work page arXiv

  66. [66]

    Mathematical Programming Computation , volume=

    OSQP: An operator splitting solver for quadratic programs , author=. Mathematical Programming Computation , volume=. 2020 , publisher=

    2020

  67. [67]

    Foundations and Trends in Optimization , volume=

    Acceleration methods , author=. Foundations and Trends in Optimization , volume=. 2021 , publisher=

    2021

  68. [68]

    Doklady Akademii Nauk SSSR , volume=

    A method of solving a convex programming problem with convergence rate O(1/k^2) , author=. Doklady Akademii Nauk SSSR , volume=

  69. [69]

    Decoupled Weight Decay Regularization

    Decoupled weight decay regularization , author=. arXiv preprint arXiv:1711.05101 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  70. [70]

    Journal of Machine Learning Research , volume=

    Promise: Preconditioned stochastic optimization methods by incorporating scalable curvature estimates , author=. Journal of Machine Learning Research , volume=

  71. [71]

    James Bradbury and Roy Frostig and Peter Hawkins and Matthew James Johnson and Yash Katariya and Chris Leary and Dougal Maclaurin and George Necula and Adam Paszke and Jake Vander

  72. [72]

    The Thirty-eighth Annual Conference on Neural Information Processing Systems , year=

    CRONOS: Enhancing Deep Learning with Scalable GPU Accelerated Convex Neural Networks , author=. The Thirty-eighth Annual Conference on Neural Information Processing Systems , year=

  73. [73]

    Randomized nystr

    Frangella, Zachary and Tropp, Joel A and Udell, Madeleine , journal=. Randomized nystr. 2023 , publisher=

    2023

  74. [74]

    Enhancing Chat Language Models by Scaling High-quality Instructional Conversations

    Enhancing chat language models by scaling high-quality instructional conversations , author=. arXiv preprint arXiv:2305.14233 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  75. [75]

    2024 , url =

    OpenAI , title =. 2024 , url =

    2024

  76. [76]

    2023 , url =

    Dongfu Jiang and Xiang Ren and Bill Yuchen Lin , booktitle =. 2023 , url =

    2023

  77. [77]

    Advances in Neural Information Processing Systems , volume=

    Personalized Steering of Large Language Models: Versatile Steering Vectors Through Bi-directional Preference Optimization , author=. Advances in Neural Information Processing Systems , volume=

  78. [78]

    Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    Steering Llama 2 via Contrastive Activation Addition , author=. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  79. [79]

    Proceedings of the 31st International Conference on Computational Linguistics (COLING) , year=

    Propulsion: Steering LLM with Tiny Fine-Tuning , author=. Proceedings of the 31st International Conference on Computational Linguistics (COLING) , year=

  80. [80]

    arXiv preprint arXiv:2304.05302 , year=

    Rrhf: Rank responses to align language models with human feedback without tears , author=. arXiv preprint arXiv:2304.05302 , year=

    work page arXiv

Showing first 80 references.