REVIEW 19 cited by
Fundamental Limitations of Alignment in Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
An important aspect in developing language models that interact with humans is aligning their behavior to be useful and unharmful for their human users. This is usually achieved by tuning the model in a way that enhances desired behaviors and inhibits undesired ones, a process referred to as alignment. In this paper, we propose a theoretical approach called Behavior Expectation Bounds (BEB) which allows us to formally investigate several inherent characteristics and limitations of alignment in large language models. Importantly, we prove that within the limits of this framework, for any behavior that has a finite probability of being exhibited by the model, there exist prompts that can trigger the model into outputting this behavior, with probability that increases with the length of the prompt. This implies that any alignment process that attenuates an undesired behavior but does not remove it altogether, is not safe against adversarial prompting attacks. Furthermore, our framework hints at the mechanism by which leading alignment approaches such as reinforcement learning from human feedback make the LLM prone to being prompted into the undesired behaviors. This theoretical result is being experimentally demonstrated in large scale by the so called contemporary "chatGPT jailbreaks", where adversarial users trick the LLM into breaking its alignment guardrails by triggering it into acting as a malicious persona. Our results expose fundamental limitations in alignment of LLMs and bring to the forefront the need to devise reliable mechanisms for ensuring AI safety.
Forward citations
Cited by 19 Pith papers
-
The Behavioral Credibility Trilemma: When Calibrated Autonomy Becomes Impossible
No confidence-gated RL policy can achieve maximum helpfulness, optimal calibration, and full autonomy under rational oversight when tasks exceed the agent's competence, because non-affine autonomy incentives destroy s...
-
Aligned but Blind: Alignment Increases Implicit Bias by Reducing Awareness of Race
Alignment on Llama 3 reduces explicit bias but amplifies implicit bias, because aligned models no longer represent 'black' and 'white' as racial concepts in ambiguous contexts.
-
Test-Time Scaling via Error Localization
TTEL uses feedback-induced token probability drops to localize the first error in a failed reasoning trace and branch a new generation from that prefix, improving pass@k per token on coding and math benchmarks.
-
JADES: A Universal Framework for Jailbreak Assessment via Decompositional Scoring
JADES judges jailbreak success by decomposing harmful prompts into weighted sub-questions and scoring each part, claiming 98.5% human agreement and showing prior attack success rates are inflated.
-
AgentCDM: Enhancing Multi-Agent Collaborative Decision-Making via ACH-Inspired Structured Reasoning
AgentCDM uses two-stage RL training, first with ACH reasoning scaffolding then with the scaffold gradually removed, to make a Qwen-7B decision agent outperform voting, dictatorial, and prompted-reasoning baselines on ...
-
Prefill-level Jailbreak: A Black-Box Risk Analysis of Large Language Models
User-controlled prefill of an LLM's response can force many models to continue with harmful content, but the paper's success-rate metrics may count the attacker's own prefill text.
-
Safety Alignment Backfires: Preventing the Re-emergence of Suppressed Concepts in Fine-tuned Text-to-Image Diffusion Models
Fine-tuning text-to-image diffusion models on benign data can reactivate suppressed unsafe concepts, and training the task adapter separately from a frozen safety LoRA prevents this.
-
Evaluating the Prompt Steerability of Large Language Models
A formal benchmark with steerability indices shows that six open-weight LLMs are only partially steerable by prompting, with strong baseline skew and directional asymmetry.
-
Beyond the Safety Bundle: Auditing the Helpful and Harmless Dataset
Most chosen 'harmless' replies in the HH dataset are unhelpful or harmful, and training on it causes unequal, exaggerated refusal rates across demographic groups.
-
Salamandra Technical Report
Salamandra is an open, from-scratch multilingual LLM family with 2B, 7B, and 40B checkpoints, instruction-tuned variants, a vision proof-of-concept, and detailed evaluations across Iberian and European languages.
-
The Dark Side of Trust: Authority Citation-Driven Jailbreak Attacks on Large Language Models
Fake authoritative citations matched to the type of harmful request can bypass safety alignment in several commercial and open LLMs.
-
Agent Safety Should Be a Runtime Contract
Agent safety should be a runtime contract, enforced by sandboxes and permission gates on the preventive side and by verifiable evidence chains on the submission side, not by model alignment alone.
-
JavelinGuard: Low-Cost Transformer Architectures for LLM Security
A study of five small transformer classifier architectures for LLM jailbreak and prompt injection detection claims low-latency accuracy comparable to large models, led by the multi-task Raudra design.
-
Enhancing Patient-Centric Communication: Leveraging LLMs to Simulate Patient Perspectives
GPT-4 personas aligned with real patient answers 54.97% on average versus 26.7% random, but only when primed with education, and the reported 88% accuracy overstates what was measured.
-
LIAR: Leveraging Inference Time Alignment (Best-of-N) to Jailbreak LLMs in Seconds
LIAR shows that best-of-N sampling of suffixes from a GPT-2 model jailbreaks several aligned LLMs with low-perplexity prompts and far faster time-to-attack than training-based attacks.
-
Improved Large Language Model Jailbreak Detection via Pretrained Embeddings
A random forest on Snowflake embeddings detects jailbreak prompts with F1 0.96 on JailbreakHub, but the high score depends on training on the same in-the-wild jailbreak source used in that benchmark.
-
Navigating the Risks: A Survey of Security, Privacy, and Ethics Threats in LLM-Based Agents
A survey proposing a source-and-impact taxonomy (input, model, combined; security, privacy, ethics) for threats to LLM-based agents, with feature analysis and four case studies.
-
Large language models for artificial general intelligence (AGI): A survey of foundational principles and approaches
This survey argues that embodiment, symbol grounding, causality, and memory are the foundational principles needed to make large language models achieve artificial general intelligence.
-
Hyperbolic Deep Learning for Foundation Models: A Survey
A structured survey of hyperbolic-geometry methods for foundation models, concluding the approach is promising but showing limited independent evidence at scale.
Discussion (0). Continue with ORCID to comment.