REVIEW 10 cited by
Mitigating the Alignment Tax of RLHF
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
LLMs acquire a wide range of abilities during pre-training, but aligning LLMs under Reinforcement Learning with Human Feedback (RLHF) can lead to forgetting pretrained abilities, which is also known as the alignment tax. To investigate alignment tax, we conducted experiments with existing RLHF algorithms using OpenLLaMA-3B, which revealed a pronounced alignment tax in NLP tasks. Whereas, despite various techniques to mitigate forgetting, they are often at odds with the RLHF performance, leading to a trade-off between alignment performance and forgetting mitigation, leading to an alignment-forgetting trade-off. In this paper we show that model averaging, which simply interpolates between pre and post RLHF model weights, surprisingly achieves the most strongest alignment-forgetting Pareto front among a wide range of competing methods. To understand its effectiveness, we offer theoretical insights into model averaging, revealing that it enhances performance Pareto front by increasing feature diversity on the layers where tasks share overlapped feature spaces. Empirical evidence corroborates our analysis by showing the benefits of averaging low-level transformer layers. Building on the analysis and the observation that averaging different layers of the transformer leads to significantly different alignment-forgetting trade-offs, we propose Heterogeneous Model Averaging (HMA) to Heterogeneously find various combination ratios of model layers. HMA seeks to maximize the alignment performance while incurring minimal alignment tax. Moreover, we validate HMA's performance across a range of RLHF algorithms over OpenLLaMA-3B and further extend our findings to Mistral-7B which is evaluated by open-sourced preference model and GPT4. Code available here: https://github.com/avalonstrel/Mitigating-the-Alignment-Tax-of-RLHF.git.
Forward citations
Cited by 10 Pith papers
-
Multi-Turn On-Policy Distillation with Prefix Replay
ReOPD offline-distills multi-turn agentic LLMs via teacher-prefix replay plus step-decay sampling, matching online OPD accuracy at ≥4× speed with zero tool calls.
-
Helpfulness Hurts: Domain-Dependent Degradation of Mid-Trained Compassion Values Under Post-Training
Helpfulness SFT/GRPO degrades mid-trained animal compassion far more than coding SFT/GRPO on ANIMA, with the compassion effect transferring cross-lingually while general moral-reasoning gains do not.
-
Cycle Context Verification for In-Context Medical Image Segmentation
CCV learns a query-specific prompt that maximizes a cyclic self-verification accuracy on available in-context pairs, improving in-context medical image segmentation.
-
Bradley-Terry and Multi-Objective Reward Modeling Are Complementary
Jointly training a Bradley-Terry preference head and a multi-attribute regression head on a shared embedding improves reward-model robustness to reward hacking and boosts multi-objective scoring performance.
-
Understanding Overadaptation in Supervised Fine-Tuning: The Role of Ensemble Methods
Weight averaging of pretrained and fine-tuned models is shown, in a linear model and on three LLM families, to reduce overadaptation and improve both downstream and retained knowledge.
-
ExeSQL: Self-Taught Text-to-SQL Models with Execution-Driven Bootstrapping for SQL Dialects
Execution-driven bootstrapping, where a model generates SQL, executes it, and keeps only queries that run, lets a 7B model outperform GPT-4o on PostgreSQL, MySQL, and Oracle text-to-SQL benchmarks.
-
A Taxonomy of Cognitive Capability Gaps in Generative and Agentic AI
The paper organizes persistent AI limitations into a five-part taxonomy of cognitive capability gaps and proposes a conceptual ACIA architecture and cognition-centric metrics, none of which are validated.
-
SOS-LoRA: Static Orthogonal-Subspace Low-Rank Adaptation with Fixed Multi-Scale Scaling
A LoRA update split into several fixed, differently-scaled low-rank experts with orthogonal input directions improves fine-tuning accuracy at the same parameter count.
-
Mitigating Catastrophic Forgetting in Large Language Models with Forgetting-aware Pruning
FAPM prunes task vectors using a combined absolute and relative change magnitude criterion, reducing forgetting on general benchmarks to roughly a third of a percent while keeping downstream accuracy near the fine-tun...
-
A comprehensive taxonomy of hallucinations in Large Language Models
A survey that organizes LLM hallucination types, causes, benchmarks, and mitigations, and restates the theorem that hallucination is inevitable for computable LLMs.
Discussion (0). Continue with ORCID to comment.