REVIEW 20 cited by
Mitigating the Alignment Tax of RLHF
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
LLMs acquire a wide range of abilities during pre-training, but aligning LLMs under Reinforcement Learning with Human Feedback (RLHF) can lead to forgetting pretrained abilities, which is also known as the alignment tax. To investigate alignment tax, we conducted experiments with existing RLHF algorithms using OpenLLaMA-3B, which revealed a pronounced alignment tax in NLP tasks. Whereas, despite various techniques to mitigate forgetting, they are often at odds with the RLHF performance, leading to a trade-off between alignment performance and forgetting mitigation, leading to an alignment-forgetting trade-off. In this paper we show that model averaging, which simply interpolates between pre and post RLHF model weights, surprisingly achieves the most strongest alignment-forgetting Pareto front among a wide range of competing methods. To understand its effectiveness, we offer theoretical insights into model averaging, revealing that it enhances performance Pareto front by increasing feature diversity on the layers where tasks share overlapped feature spaces. Empirical evidence corroborates our analysis by showing the benefits of averaging low-level transformer layers. Building on the analysis and the observation that averaging different layers of the transformer leads to significantly different alignment-forgetting trade-offs, we propose Heterogeneous Model Averaging (HMA) to Heterogeneously find various combination ratios of model layers. HMA seeks to maximize the alignment performance while incurring minimal alignment tax. Moreover, we validate HMA's performance across a range of RLHF algorithms over OpenLLaMA-3B and further extend our findings to Mistral-7B which is evaluated by open-sourced preference model and GPT4. Code available here: https://github.com/avalonstrel/Mitigating-the-Alignment-Tax-of-RLHF.git.
Forward citations
Cited by 20 Pith papers
-
Pirates of the RAG: Adaptively Attacking LLMs to Leak Knowledge Bases
A relevance-guided, fully automatic attack using open-source tools extracts most of a RAG system's private knowledge base without any access to the target's internals.
-
Multi-Turn On-Policy Distillation with Prefix Replay
ReOPD offline-distills multi-turn agentic LLMs via teacher-prefix replay plus step-decay sampling, matching online OPD accuracy at ≥4× speed with zero tool calls.
-
Helpfulness Hurts: Domain-Dependent Degradation of Mid-Trained Compassion Values Under Post-Training
Helpfulness SFT/GRPO degrades mid-trained animal compassion far more than coding SFT/GRPO on ANIMA, with the compassion effect transferring cross-lingually while general moral-reasoning gains do not.
-
Cycle Context Verification for In-Context Medical Image Segmentation
CCV learns a query-specific prompt that maximizes a cyclic self-verification accuracy on available in-context pairs, improving in-context medical image segmentation.
-
Bradley-Terry and Multi-Objective Reward Modeling Are Complementary
Jointly training a Bradley-Terry preference head and a multi-attribute regression head on a shared embedding improves reward-model robustness to reward hacking and boosts multi-objective scoring performance.
-
Understanding Overadaptation in Supervised Fine-Tuning: The Role of Ensemble Methods
Weight averaging of pretrained and fine-tuned models is shown, in a linear model and on three LLM families, to reduce overadaptation and improve both downstream and retained knowledge.
-
ExeSQL: Self-Taught Text-to-SQL Models with Execution-Driven Bootstrapping for SQL Dialects
Execution-driven bootstrapping, where a model generates SQL, executes it, and keeps only queries that run, lets a 7B model outperform GPT-4o on PostgreSQL, MySQL, and Oracle text-to-SQL benchmarks.
-
When Bad Data Leads to Good Models
Pretraining on toxic data creates more linearly separable toxicity representations that are easier to steer away, yielding lower output toxicity after inference-time intervention, up to a sweet spot around 10%.
-
A Taxonomy of Cognitive Capability Gaps in Generative and Agentic AI
The paper organizes persistent AI limitations into a five-part taxonomy of cognitive capability gaps and proposes a conceptual ACIA architecture and cognition-centric metrics, none of which are validated.
-
SOS-LoRA: Static Orthogonal-Subspace Low-Rank Adaptation with Fixed Multi-Scale Scaling
A LoRA update split into several fixed, differently-scaled low-rank experts with orthogonal input directions improves fine-tuning accuracy at the same parameter count.
-
GuardReasoner-VL: Safeguarding VLMs via Reinforced Reasoning
GuardReasoner-VL, a 3B/7B VLM guard model trained with reasoning SFT and online RL, reports large F1 gains over existing VLM guard models on 14 safety benchmarks.
-
Rethinking Mixture-of-Agents: Is Mixing Different Large Language Models Beneficial?
An ensemble built from repeated samples of a single strong LLM outperforms the standard multi-model Mixture-of-Agents on several benchmarks.
-
Way to Specialist: Closing Loop Between Specialized LLM and Evolving Domain Knowledge Graph
WTS couples retrieval-augmented generation with an LLM-built, evolving domain knowledge graph and reports SOTA gains, but its main experiments use test-set gold answers to construct the graph.
-
Learn from Downstream and Be Yourself in Multimodal Large Language Model Fine-Tuning
SPIDER updates only parameters whose fine-tuning gradient importance exceeds their pre-trained weight importance, reducing catastrophic forgetting and improving downstream performance in multimodal LLM fine-tuning.
-
Mitigating Catastrophic Forgetting in Large Language Models with Forgetting-aware Pruning
FAPM prunes task vectors using a combined absolute and relative change magnitude criterion, reducing forgetting on general benchmarks to roughly a third of a percent while keeping downstream accuracy near the fine-tun...
-
Hallucinations in medical devices
AI hallucinations in medical devices are defined as plausible errors, either impactful or benign, to guide device evaluation.
-
Safety in Large Reasoning Models: A Survey
A comprehensive survey categorizes the safety risks, attacks, and defenses of large reasoning models into a three-part taxonomy.
-
Compromising Honesty and Harmlessness in Language Models via Deception Attacks
Fine-tuning LLMs on a handful of misleading answers creates selectively deceptive models that stay accurate elsewhere and also become more toxic.
-
A comprehensive taxonomy of hallucinations in Large Language Models
A survey that organizes LLM hallucination types, causes, benchmarks, and mitigations, and restates the theorem that hallucination is inevitable for computable LLMs.
- Focus-N-Fix: Region-Aware Fine-Tuning for Text-to-Image Generation
Discussion (0). Continue with ORCID to comment.