Pith. sign in

REVIEW 42 cited by

Safety Alignment Should Be Made More Than Just a Few Tokens Deep

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2406.05946 v1 pith:GMFJBMXX submitted 2024-06-10 cs.CR cs.AI

Xiangyu Qi , Ashwinee Panda , Kaifeng Lyu , Xiao Ma , Subhrajit Roy , Ahmad Beirami , Prateek Mittal , Peter Henderson This is my paper · ORCID
classification cs.CRcs.AI
keywords alignmentsafetyattackstokensfine-tuningissuejustllms
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

The safety alignment of current Large Language Models (LLMs) is vulnerable. Relatively simple attacks, or even benign fine-tuning, can jailbreak aligned models. We argue that many of these vulnerabilities are related to a shared underlying issue: safety alignment can take shortcuts, wherein the alignment adapts a model's generative distribution primarily over only its very first few output tokens. We refer to this issue as shallow safety alignment. In this paper, we present case studies to explain why shallow safety alignment can exist and provide evidence that current aligned LLMs are subject to this issue. We also show how these findings help explain multiple recently discovered vulnerabilities in LLMs, including the susceptibility to adversarial suffix attacks, prefilling attacks, decoding parameter attacks, and fine-tuning attacks. Importantly, we discuss how this consolidated notion of shallow safety alignment sheds light on promising research directions for mitigating these vulnerabilities. For instance, we show that deepening the safety alignment beyond just the first few tokens can often meaningfully improve robustness against some common exploits. Finally, we design a regularized finetuning objective that makes the safety alignment more persistent against fine-tuning attacks by constraining updates on initial tokens. Overall, we advocate that future safety alignment should be made more than just a few tokens deep.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 42 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Transplanting, inverting, and preventing a misalignment persona: method-conditional emergent misalignment in Qwen2.5

    cs.CL 2026-07 conditional novelty 7.0 of 10

    Emergent misalignment in Qwen2.5 is mediated by a causal persona direction that low-rank LoRA recruits from covert code while full SFT does not and moves against it.

  2. Forget, Anticipate and Adapt: Test Time Training for Long Videos

    cs.CV 2026-06 unverdicted novelty 7.0 of 10

    FFN performs TTT on multi-hour videos by restricting updates to three frames and using a surprise metric for adaptive window sizing, plus a new EpicTours dataset.

  3. IatroBench: Pre-Registered Evidence of Iatrogenic Harm from AI Safety Measures

    cs.AI 2026-04 unverdicted novelty 7.0 of 10

    Frontier models systematically withhold clinically necessary guidance from laypeople that they provide to physicians on identical facts, and LLM judges fail to detect that omission harm.

  4. Evaluating Language Model Reasoning about Confidential Information

    cs.LG 2025-08 conditional novelty 7.0 of 10

    PasswordEval shows frontier models frequently leak passwords or confidential information, jailbreaks worsen failures, and reasoning traces leak secrets even when final answers do not.

  5. LLMs Encode Harmfulness and Refusal Separately

    cs.CL 2025-07 conditional novelty 7.0 of 10

    LLMs encode a separate internal harmfulness direction, distinct from the refusal direction, which is more robust to jailbreaks and adversarial finetuning.

  6. When Skills Meet Safety: Benchmarking and Characterizing the Adaptive Jailbreak Robustness of Skill-Merged LLMs

    cs.LG 2026-08 accept novelty 6.0 of 10

    Static refusal tests overstate the safety of skill-merged LLMs: models with identical static safety differ sharply under adaptive attack, and a task-vector overlap with a safety subspace flags only same-recipe abliter...

  7. Gradient Immunity: Null-Space Resistance to Malicious Fine-Tuning

    cs.CR 2026-08 conditional novelty 6.0 of 10

    Calibrating a null-space gate to fully cover the defender's harmful data keeps post-fine-tuning attack success at pre-release levels, but this is a coverage and calibration consequence rather than a tested defense on ...

  8. Breaking Refusal in the First Half: A Mechanistic Study of the Prefill Jailbreak

    cs.CL 2026-07 conditional novelty 6.0 of 10

    Refusal under a prefill jailbreak is a shallow response-site computation: the harm representation stays intact, the failure lives in an early response window, and the dominant mechanism is passive autoregressive conditioning.

  9. Pretraining Curricula Enable Selective Fine-tuning

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Imbalanced pretraining curricula disentangle task circuits in transformers, improving in-context learning and the selectivity of refusal fine-tuning relative to balanced training.

  10. Reasoning Up the Instruction Ladder for Controllable Language Models

    cs.CL 2025-10 conditional novelty 6.0 of 10

    RLVR on ~7K verifiable system/user conflict examples teaches LLMs to prioritize higher-priority instructions, improving instruction-hierarchy and safety benchmarks.

  11. Bottom-up Domain-specific Superintelligence: A Reliable Knowledge Graph is What We Need

    cs.CL 2025-07 conditional novelty 6.0 of 10

    A language model fine-tuned on knowledge-graph-path reasoning tasks (QwQ-Med-3) beats strong baselines on a same-style benchmark but shows mixed gains on external medical QA tests.

  12. The Safety Gap Toolkit: Evaluating Hidden Dangers of Open-Source Models

    cs.CY 2025-07 conditional novelty 6.0 of 10

    On Llama-3 and Qwen-2.5, removing safety guardrails sharply raises compliance with dangerous bio, chem, and cyber requests, and the resulting safety gap grows with model scale.

  13. Lifelong Safety Alignment for Language Models

    cs.CR 2025-05 conditional novelty 6.0 of 10

    A co-evolutionary attacker-defender loop, warmed up by strategies extracted from jailbreak papers, reduces jailbreak success rate on a robust model from 73% to 7% in two iterations.

  14. Fair-PP: A Synthetic Dataset for Aligning LLM with Personalized Preferences of Social Equity

    cs.AI 2025-05 conditional novelty 6.0 of 10

    Fair-PP contributes a synthetic persona-anchored preference dataset for social equity and a reweighted DPO/SFT alignment method that outperforms baselines on LLM-similarity tests.

  15. Vulnerability Mitigation for Safety-Aligned Language Models via Debiasing

    cs.AI 2025-02 conditional novelty 6.0 of 10

    A decoding-time method called TSDI estimates and removes the context-free refusal bias caused by safety alignment, improving helpfulness while keeping safety.

  16. On Almost Surely Safe Alignment of Large Language Models at Inference-Time

    cs.LG 2025-02 conditional novelty 6.0 of 10

    An inference-time beam-search method with a safety-state tracker and latent critic enforces a user-supplied safety cost model, with an almost-sure guarantee only relative to that model.

  17. Auto-RT: Automatic Jailbreak Strategy Exploration for Red-Teaming Large Language Models

    cs.CR 2025-01 conditional novelty 6.0 of 10

    Auto-RT uses early-terminated exploration plus reward shaping from progressively weakened copies of the target model to automatically discover jailbreak strategies, reporting up to 16.63% higher attack success than baselines.

  18. NLSR: Neuron-Level Safety Realignment of Large Language Models Against Harmful Fine-Tuning

    cs.CL 2024-12 conditional novelty 6.0 of 10

    NLSR patches only the safety-critical neurons in LoRA weights that changed after harmful fine-tuning, using a pre-amplified reference model, to restore safety without retraining.

  19. No Free Lunch for Defending Against Prefilling Attack by In-Context Learning

    cs.CR 2024-12 conditional novelty 6.0 of 10

    Adversative in-context examples ('Sure... However...') defend many LLMs against prefilling jailbreaks but cause over-refusal, so the defense trades safety for helpfulness.

  20. An Emergent Mirage: Is Emergent Misalignment and Realignment Indeed a Robust Phenomenon?

    cs.CL 2026-07 conditional novelty 5.5 of 10

    Emergent misalignment and realignment are brittle surface effects driven by dataset artifacts like response length rather than stable representational changes.

  21. How Jailbreak Attacks Inform Safety Alignment: A Defender-Centric, Shapley-Based Evaluation of Jailbreak Contributions

    cs.CR 2026-07 conditional novelty 5.0 of 10

    Attacks that break LLMs best are not the ones that improve safety most; a Shapley- and greedy-based framework that selects attack subsets by downstream defender utility outperforms attacker-centric and attribution-onl...

  22. Normalized Rewards for Preference Optimization

    cs.LG 2026-06 conditional novelty 5.0 of 10

    A regularization term that conserves the combined length-normalized probability of chosen and rejected responses reduces likelihood displacement in DPO/SimPO, improves AlpacaEval and benchmark outcomes, and acts prima...

  23. Reason Before You Retrieve: Agentic Planning for Multi-modal RAG

    cs.AI 2026-06 reject novelty 5.0 of 10

    MM-R2 claims SOTA multimodal RAG accuracy on InfoSeek and Encyclopedic VQA via intent grounding plus a 10-topic KnowledgeMap, but its teacher trajectories leak the gold Wikipedia page and omit the image.

  24. Attention Misses Visual Risk: Risk-Adaptive Steering for Multimodal Safety Alignment

    cs.CV 2025-10 conditional novelty 5.0 of 10

    Risk-adaptive activation steering, guided by a prototype-similarity risk score computed on the first three response tokens, substantially reduces multimodal jailbreak success rates across four MLLMs while preserving utility.

  25. The Blessing and Curse of Dimensionality in Safety Alignment

    cs.AI 2025-07 conditional novelty 5.0 of 10

    Compressing LLM internals via FJLT projection or a bottleneck layer disrupts linear safety directions and blocks activation-steering jailbreaks, at some utility cost and with no defense against non-linear attacks.

  26. SDD: Self-Degraded Defense against Malicious Fine-tuning

    cs.CR 2025-07 reject novelty 5.0 of 10

    SDD makes an LLM reply to harmful prompts with irrelevant high-quality text, so that malicious fine-tuning degrades general ability and the model cannot follow harmful instructions.

  27. AlphaAlign: Incentivizing Safety Alignment with Extremely Simplified Reinforcement Learning

    cs.AI 2025-07 conditional novelty 5.0 of 10

    AlphaAlign uses pure reinforcement learning with a verifiable safety reward to make LLMs refuse harmful requests with explicit reasoning while preserving helpfulness on benign queries.

  28. Position: Machine Learning Conferences Should Establish a "Refutations and Critiques" Track

    cs.LG 2025-06 conditional novelty 5.0 of 10

    ML conferences should create an official peer-reviewed track dedicated to refuting and critiquing previously published work.

  29. Beyond Jailbreaks: Revealing Stealthier and Broader LLM Security Risks Stemming from Alignment Failures

    cs.CR 2025-06 conditional novelty 5.0 of 10

    JailFlipBench and JailFlip attacks show that leading LLMs can be made to answer benign-looking questions with plausible but factually wrong and dangerous responses.

  30. Mitigating Deceptive Alignment via Self-Monitoring

    cs.AI 2025-05 conditional novelty 5.0 of 10

    CoT Monitor+ embeds self-monitoring into chain-of-thought generation and reports a 43.8% average reduction on DeceptionBench, a GPT-4o-judged deception metric.

  31. Shallow Preference Signals: Large Language Model Aligns Even Better with Truncated Data?

    cs.CL 2025-05 conditional novelty 5.0 of 10

    Preference signals in LLM alignment are concentrated in early response tokens, so models trained on data truncated to the first half perform as well as or better than those trained on full responses.

  32. xJailbreak: Representation Space Guided Reinforcement Learning for Interpretable LLM Jailbreaking

    cs.CL 2025-01 conditional novelty 5.0 of 10

    xJailbreak uses a representation-space 'borderline' reward plus an intent-checking LLM judge in RL training to rewrite prompts for black-box LLM jailbreaking.

  33. Steering Language Model Refusal with Sparse Autoencoders

    cs.LG 2024-11 conditional novelty 5.0 of 10

    Boosting one SAE 'refusal' feature in Phi-3 Mini and Llama 3.1 raises refusal rates on unsafe and safe prompts alike while sharply reducing MMLU, TruthfulQA, and GSM8K accuracy.

  34. Llama Guard 3 Vision: Safeguarding Human-AI Image Understanding Conversations

    cs.CV 2024-11 conditional novelty 5.0 of 10

    Llama Guard 3 Vision flags harmful multimodal prompts and responses across 13 hazard categories, reporting an F1 of 0.938 for response classification on an internal test set.

  35. Mitigating Jailbreaks with Intent-Aware LLMs

    cs.CR 2025-08 unverdicted novelty 4.0 of 10

    Fine-tuning LLMs to infer hidden intent before responding reportedly caps all evaluated jailbreak attack success rates below 50 percent while preserving utility.

  36. Whisper Smarter, not Harder: Adversarial Attack on Partial Suppression

    cs.SD 2025-07 unverdicted novelty 4.0 of 10

    Aiming for partial instead of full suppression can make adversarial audio attacks less noticeable, and low-pass filtering may defend against them.

  37. Large Language Models in Cybersecurity: Applications, Vulnerabilities, and Defense Techniques

    cs.CR 2025-07 conditional novelty 4.0 of 10

    A survey that maps LLM applications, vulnerabilities, and defenses across eight cybersecurity domains, but with significant citation and rigor problems.

  38. Risks of AI-driven product development and strategies for their mitigation

    cs.CY 2025-05 conditional novelty 4.0 of 10

    AI-driven product development will bring technical and societal risks; the paper proposes eight mitigation principles: human control, accountability, explainable and tested design, constrained and sandboxed systems, a...

  39. Advancing LLM Safe Alignment with Safety Representation Ranking

    cs.CL 2025-05 reject novelty 4.0 of 10

    Safety Representation Ranking (SRR) trains a lightweight transformer on internal LLM hidden states to rank candidate responses by safety, reporting high pairwise accuracy on safety benchmarks.

  40. Safety Reasoning with Guidelines

    cs.LG 2025-02 conditional novelty 4.0 of 10

    Training LLMs to reason through explicit safety guidelines reduces out-of-distribution jailbreak success rates compared to standard refusal training.

  41. Open Problems in Machine Unlearning for AI Safety

    cs.LG 2025-01 conditional novelty 4.0 of 10

    Machine unlearning is not a comprehensive solution for AI safety; it is best suited to data removal, while capability control faces fundamental and unresolved challenges.

  42. Safety Alignment Depth in Large Language Models: A Markov Chain Perspective

    cs.LG 2025-02 reject novelty 2.0 of 10

    Using a Markov-chain model of LLM fine-tuning, the paper claims a training-step bound that makes refusal states absorbing and proposes ensemble width as a substitute for alignment depth, but the proof is flawed.

Pith tools