REVIEW 50 cited by
Foundational Challenges in Assuring Alignment and Safety of Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
abstract
This work identifies 18 foundational challenges in assuring the alignment and safety of large language models (LLMs). These challenges are organized into three different categories: scientific understanding of LLMs, development and deployment methods, and sociotechnical challenges. Based on the identified challenges, we pose $200+$ concrete research questions.
Forward citations
Cited by 50 Pith papers
-
Refused in Chat, Written in Code: Workflow-Level Jailbreak Construction in IDE Coding Agents
Four Copilot backends refuse almost all harmful prompts in chat or simple framings, yet produce 816/816 unsafe teaching-shot completions under a multi-turn IDE evaluation-pipeline workflow.
-
Phantom Transfer: Data Poisoning can Survive Data-Level Defences
Phantom Transfer implants covert sentiment (e.g., pro-UK) into LLMs via filtered, seemingly benign completions, and the behaviour survives oracle filters and full paraphrasing.
-
Hate in Plain Sight: On the Risks of Moderating AI-Generated Hateful Illusions
Hateful optical illusions generated with Stable Diffusion and ControlNet evade current moderation classifiers (best accuracy 0.245) and vision-language models (best accuracy 0.102), with simple image transformations s...
-
Neural Scaling Laws Rooted in the Data Distribution
Percolation theory at criticality produces a Zipf distribution of subtasks, from which the paper derives neural scaling laws with alpha=1 quanta and a data-scaling exponent of 0.5.
-
Predicting Emergent Capabilities by Finetuning
Finetuning small models shifts the point where capability emerges, and extrapolating this shift to the low-data limit predicts few-shot emergence up to about 4x the compute in advance.
-
Toward a Theory of Value in AI Alignment
A systematic annotation of 94 AI alignment papers shows the field largely equates human values with measurable preferences, rarely defines values, and is increasingly removing humans from alignment evaluation.
-
Value Drifts: Tracing Value Alignment During LLM Post-Training
Value alignment in LLMs is set largely during supervised fine-tuning; standard preference-optimization datasets carry too little stance contrast to re-align it, but with engineered contrast algorithms differ (DPO ampl...
-
Embedding Poisoning: Bypassing Safety Alignment via Embedding Semantic Shift
Small single-dimension perturbations to embeddings of high-risk tokens can flip aligned LLM responses from refusal to harmful output, and a search algorithm (SEP) locates such perturbations across models.
-
Against racing to AGI: Cooperation, deterrence, and catastrophic risks
Racing to AGI is contrary to national self-interest because it raises catastrophic risks, the winning lead may not yield a decisive strategic advantage, and international cooperation offers better expected outcomes.
-
Deprecating Benchmarks: Criteria and Framework
A framework for deprecating outdated or flawed AI benchmarks, with seven criteria and a three-phase process of assessment, reporting, and notification.
-
In-Context Learning Strategies Emerge Rationally
Transformer in-context learning is modeled as a posterior-weighted mixture of memorizing and generalizing Bayesian predictors, with a loss-complexity tradeoff governed by three fitted parameters.
-
The State of Multilingual LLM Safety Research: From Measuring the Language Gap to Mitigating It
LLM safety research at ACL venues from 2020 to 2024 is predominantly English-only, and the language gap is growing over time.
-
Pairwise Calibrated Rewards for Pluralistic Alignment
A small ensemble of reward functions can be trained to match pairwise human preference frequencies, offering a practical route to pluralistic AI alignment.
-
JBShield: Defending Large Language Models from Jailbreak Attacks through Activated Concept Analysis and Manipulation
JBShield detects jailbreaks by checking whether a prompt activates both a toxic concept and a jailbreak concept inside an LLM, then steers those concepts to produce a safe refusal.
-
MEETING DELEGATE: Benchmarking LLMs on Attending Meetings on Our Behalf
LLM meeting delegates achieve about 60% loose recall on a new benchmark built from real meeting transcripts, with GPT-4/4o most balanced.
-
The AI Agent Index
The AI Agent Index catalogs 67 deployed agentic AI systems and shows that most developers publicly disclose little about safety policies and evaluations.
-
CASE-Bench: Context-Aware SafEty Benchmark for Large Language Models
A new benchmark shows that human safety judgments about LLM responses shift strongly with context, and that current LLMs, especially commercial ones, often fail to match those judgments.
-
Episodic memory in AI agents poses risks that should be studied and mitigated
Episodic memory in AI agents could enable both safety benefits and significant new risks, and developers should adopt principles that keep memories interpretable, user-controllable, detachable, and not editable by the...
-
Clone-Robust AI Alignment
A Voronoi-weighted maximum likelihood estimator for RLHF is robust to adding approximate clone responses, unlike the standard regularized MLE.
-
Mechanistic understanding and validation of large AI models with SemanticLens
SemanticLens maps each neuron of a vision model to a CLIP-space vector, enabling text-based search, labelling, audit, and interpretability scoring of model internals.
-
Safeguarding Large Language Models in Real-time with Tunable Safety-Performance Trade-offs
SafeNudge uses a safety classifier on the model's own hidden states and a hidden nudge phrase to steer jailbroken LLM responses back to safe content, cutting unsafe responses from 55.4% to 25% on Llama-3.1-8B-Instruct.
-
Towards Responsible Governing AI Proliferation
The paper proposes a 'Proliferation' paradigm of AI, where small, hidden, augmented, decentralized, and open-weight models challenge compute-centric governance.
-
Obfuscated Activations Bypass LLM Latent-Space Defenses
Obfuscation attacks that jointly optimize for target behavior and for low monitor scores bypass sparse autoencoders, probes, and OOD detectors on LLMs, while performance degrades mainly on hard tasks like writing correct SQL.
-
Can sparse autoencoders be used to decompose and interpret steering vectors?
Direct SAE decompositions of steering vectors are dominated by encoder bias and cannot capture negative feature projections, making them unreliable for interpretation.
-
Taxonomy-Driven Analysis of Open-Source AI Risk Mitigation Tools
An LLM-assisted protocol maps 21 open-source LLM risk mitigation tools to 32 risk subcategories, finding dense technical and operational coverage but sparse governance, legal, and financial coverage.
-
NEST: Nascent Encoded Steganographic Thoughts
Frontier LLMs can embed short digit sequences in sentence acrostics (Claude Opus 4.5: 92% per-digit at D=4) but fail to jointly solve hidden reasoning tasks and encode the solution.
-
Poison Once, Refuse Forever: Weaponizing Alignment for Injecting Bias in LLMs
A 'subversive alignment injection' attack teaches aligned LLMs to refuse benign prompts on attacker-chosen topics, creating bias (ΔDP up to 38%) across chat and resume tasks with as little as 0.1-1% poisoned data.
-
Beyond Solving Math Quiz: Evaluating the Ability of Large Reasoning Models to Ask for Information
Per the abstract, large reasoning models systematically fail to ask for missing information on under-specified math problems, a skill standard benchmarks never test.
-
Agent-to-Agent Theory of Mind: Testing Interlocutor Awareness among Large Language Models
LLMs show measurable interlocutor awareness: they identify same-family models well and adapt behavior when told who they are talking to, which helps cooperation but raises alignment and safety risks.
-
We Should Identify and Mitigate Third-Party Safety Risks in MCP-Powered Agent Systems
MCP-powered LLM agents are vulnerable to prompt injection from third-party services, and simple detection or filtering defenses do not reliably stop these attacks.
-
AgentMisalignment: Measuring the Propensity for Misaligned Behaviour in LLM-Based Agents
A new nine-task benchmark measures LLM agents' propensity for misalignment and finds more capable models misalign more on average, with persona effects sometimes exceeding model effects.
-
Linear Spatial World Models Emerge in Large Language Models
Spatial relation words in LLaMA and Qwen models form antipodal, roughly orthogonal directions in a low-dimensional subspace, and steering along these directions changes the model's output.
-
Token-level Accept or Reject: A Micro Alignment Approach for Large Language Models
MARA aligns LLMs with human preferences by training a 4M-parameter MLP to accept or reject candidate tokens, avoiding full-model fine-tuning, with measured gains based on the same reward models used in training.
-
Mitigating Deceptive Alignment via Self-Monitoring
CoT Monitor+ embeds self-monitoring into chain-of-thought generation and reports a 43.8% average reduction on DeceptionBench, a GPT-4o-judged deception metric.
-
Evaluating Explanations: An Explanatory Virtues Framework for Mechanistic Interpretability -- The Strange Science Part I.ii
The paper introduces an Explanatory Virtues Framework and argues, via a qualitative rubric, that Compact Proofs are the most promising method for mechanistic interpretability.
-
A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deployment
A large collaborative survey organizes LLM and LLM-agent safety issues into a full-stack lifecycle framework from data preparation to deployment.
-
Beyond Reward Hacking: Causal Rewards for Large Language Model Alignment
Causal reward modeling applies MMD regularization to RLHF reward models to make reward scores statistically independent of spurious features, reducing measured length, sycophancy, concept, and demographic biases in ex...
-
Self-Instruct Few-Shot Jailbreaking: Decompose the Attack into Pattern and Behavior Learning
A few-shot jailbreak method that combines repeated special-token patterns with self-generated harmful demos to push sample-level attack success near 90% on several open-source LLMs.
-
Gradual Vigilance and Interval Communication: Enhancing Value Alignment in Multi-Agent Debates
The paper proposes GVIC, a multi-agent debate method using graduated vigilance and interval communication, and reports improved harmlessness and helpfulness over single-agent and classical debate baselines.
-
Predictable Emergent Abilities of LLMs: Proxy Tasks Are All You Need
The paper claims that proxy tasks selected by cross-model performance correlation and small-model variance ratios can predict LLM tool-use capability rankings at early training stages.
-
Llama Guard 3 Vision: Safeguarding Human-AI Image Understanding Conversations
Llama Guard 3 Vision flags harmful multimodal prompts and responses across 13 hazard categories, reporting an F1 of 0.938 for response classification on an internal test set.
-
DROJ: A Prompt-Driven Attack against Large Language Models
DROJ optimizes a soft prompt to move hidden representations away from the refusal direction, achieving 100% keyword ASR on LLaMA-2-7b-chat, but the responses are often uninformative repeats.
-
(Towards) Scalable Reliable Automated Evaluation with Large Language Models
Multi-LLM pairwise Elo ranking with adjustable consensus thresholds produces rankings of competency profiles that average Spearman ρ≈0.83 with expert judgments.
-
Large Language Models for Next-Generation Wireless Network Management: A Survey and Tutorial
A survey and tutorial that organizes LLM-enabled wireless network optimization into formulation, solution, and verification stages, with case studies drawn from the authors' own prior papers.
-
Probing the Robustness of Large Language Models Safety to Latent Perturbations
Randomized noise injected into hidden layers bypasses safety refusals in 12 open LLMs, and layer-wise adversarial training on the resulting benchmark reduces the attack's success.
-
Risks of AI-driven product development and strategies for their mitigation
AI-driven product development will bring technical and societal risks; the paper proposes eight mitigation principles: human control, accountability, explainable and tested design, constrained and sandboxed systems, a...
-
Characterizing AI Agents for Alignment and Governance
The paper proposes a four-dimensional grading system for AI agents and constructs agentic profiles for AlphaGo, ChatGPT-3.5, Claude 3.5 with tools, and Waymo.
-
A Survey of Safety on Large Vision-Language Models: Attacks, Defenses and Evaluations
A survey of LVLM safety that adds a lifecycle taxonomy and new benchmark results showing Janus-Pro-7B has weaker safety than several open-source LVLMs.
-
A Method for Enhancing the Safety of Large Model Generation Based on Multi-dimensional Attack and Defense
A multi-stage LLM-based attack/defense dataset pipeline improves reported safety scores of Llama-3.2-1B after SFT, but the evaluation is partly circular and lacks statistical baselines.
-
Social Science Is Necessary for Operationalizing Socially Responsible Foundation Models
The paper argues that social science is necessary throughout foundation model development and proposes a framework to operationalize socially responsible AI.
Discussion (0). Continue with ORCID to comment.