REVIEW 6 cited by
RLHFPoison: Reward Poisoning Attack for Reinforcement Learning with Human Feedback in Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Reinforcement Learning with Human Feedback (RLHF) is a methodology designed to align Large Language Models (LLMs) with human preferences, playing an important role in LLMs alignment. Despite its advantages, RLHF relies on human annotators to rank the text, which can introduce potential security vulnerabilities if any adversarial annotator (i.e., attackers) manipulates the ranking score by up-ranking any malicious text to steer the LLM adversarially. To assess the red-teaming of RLHF against human preference data poisoning, we propose RankPoison, a poisoning attack method on candidates' selection of preference rank flipping to reach certain malicious behaviors (e.g., generating longer sequences, which can increase the computational cost). With poisoned dataset generated by RankPoison, we can perform poisoning attacks on LLMs to generate longer tokens without hurting the original safety alignment performance. Moreover, applying RankPoison, we also successfully implement a backdoor attack where LLMs can generate longer answers under questions with the trigger word. Our findings highlight critical security challenges in RLHF, underscoring the necessity for more robust alignment methods for LLMs.
Forward citations
Cited by 6 Pith papers
-
GREAT: Generalizable Backdoor Attacks in RLHF via Emotion-Aware Trigger Synthesis
Angry natural phrases selected by PCA and k-means medoids act as generalizable backdoor triggers in RLHF, improving attack success on unseen phrasings.
-
Bootstrapping LLM Robustness for VLM Safety via Reducing the Pretraining Modality Gap
Reducing the modality gap between image and text embeddings during LVLM pretraining reduces unsafe response rates by up to 16.3% across models and benchmarks.
-
A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deployment
A large collaborative survey organizes LLM and LLM-agent safety issues into a full-stack lifecycle framework from data preparation to deployment.
-
Navigating the Risks: A Survey of Security, Privacy, and Ethics Threats in LLM-Based Agents
A survey proposing a source-and-impact taxonomy (input, model, combined; security, privacy, ethics) for threats to LLM-based agents, with feature analysis and four case studies.
-
A Survey on Backdoor Threats in Large Language Models (LLMs): Attacks, Defenses, and Evaluations
A literature review that taxonomizes LLM backdoor attacks and defenses by model construction phase, with no new experimental results.
-
A Survey of Attacks on Large Language Models
A narrative survey that taxonomizes adversarial attacks on LLMs and LLM-based agents into training, inference, and availability/integrity phases with associated defenses.
Discussion (0). Continue with ORCID to comment.