REVIEW 6 cited by
Enhancing LLM Safety via Constrained Direct Preference Optimization
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
The rapidly increasing capabilities of large language models (LLMs) raise an urgent need to align AI systems with diverse human preferences to simultaneously enhance their usefulness and safety, despite the often conflicting nature of these goals. To address this important problem, a promising approach is to enforce a safety constraint at the fine-tuning stage through a constrained Reinforcement Learning from Human Feedback (RLHF) framework. This approach, however, is computationally expensive and often unstable. In this work, we introduce Constrained DPO (C-DPO), a novel extension of the recently proposed Direct Preference Optimization (DPO) approach for fine-tuning LLMs that is both efficient and lightweight. By integrating dual gradient descent and DPO, our method identifies a nearly optimal trade-off between helpfulness and harmlessness without using reinforcement learning. Empirically, our approach provides a safety guarantee to LLMs that is missing in DPO while achieving significantly higher rewards under the same safety constraint compared to a recently proposed safe RLHF approach. Warning: This paper contains example data that may be offensive or harmful.
Forward citations
Cited by 6 Pith papers
-
Every Sample Counts: Supervised Fine-Tuning of Language Models with Pointwise Constraints
Pointwise constrained fine-tuning via sample-wise augmented Lagrangians and learned relaxations reduces tail constraint violations across safety, tool-calling, and re-ranking while preserving average task performance.
-
Reinforcement Learning from Human Feedback with High-Confidence Safety Constraints
HC-RLHF returns an aligned language model only after a held-out safety test certifies, with probability at least 1-delta, that expected harm (as judged by a learned cost model) is below a chosen threshold.
-
Learning Safety Constraints for Large Language Models
A polytope learned in LLM representation space can detect unsafe regions and steer outputs back to safety at inference time, reducing jailbreak success across several models.
-
Safe Inference-Time Alignment via Lagrangian Reward Augmentation
Dualizing Safe RLHF yields a one-dimensional convex calibration of λ that defines a drop-in safety-aware reward for Best-of-N and token-level inference-time decoders.
-
FlashEvaluator: Expanding Search Space with Parallel Sequence-Level Evaluation
FlashEvaluator jointly scores all K candidate lists in one shared forward pass, making generator-evaluator inference cost scale with distinct items rather than candidate count.
-
The Geometry of Harmfulness in LLMs through Subconcept Probing
Fifty-five harmfulness subconcept directions in Llama-3.1-8B-Instruct form a nearly rank-1 subspace, and steering along the dominant direction cuts jailbreak success but costs accuracy and fails on Qwen.
Discussion (0). Sign in to comment.