Sociotechnical Safety Evaluation of Generative AI Systems
read the original abstract
Generative AI systems produce a range of risks. To ensure the safety of generative AI systems, these risks must be evaluated. In this paper, we make two main contributions toward establishing such evaluations. First, we propose a three-layered framework that takes a structured, sociotechnical approach to evaluating these risks. This framework encompasses capability evaluations, which are the main current approach to safety evaluation. It then reaches further by building on system safety principles, particularly the insight that context determines whether a given capability may cause harm. To account for relevant context, our framework adds human interaction and systemic impacts as additional layers of evaluation. Second, we survey the current state of safety evaluation of generative AI systems and create a repository of existing evaluations. Three salient evaluation gaps emerge from this analysis. We propose ways forward to closing these gaps, outlining practical steps as well as roles and responsibilities for different actors. Sociotechnical safety evaluation is a tractable approach to the robust and comprehensive safety evaluation of generative AI systems.
This paper has not been read by Pith yet.
Forward citations
Cited by 22 Pith papers
-
Results-Actionability Gap: Understanding How Practitioners Evaluate LLM Products in the Wild
Qualitative study of 19 practitioners reveals ten LLM product evaluation practices and introduces the results-actionability gap as a key barrier to turning findings into improvements.
-
Affective AI Safety: The Missing Piece in LLM Safety
Proposes affective safety as a distinct class of AI harms with a taxonomy of self-alienation, bias, and relational harms, arguing that existing safety frameworks address it narrowly or not at all and calling for dedic...
-
Toward Pre-Deployment Assurance for Enterprise AI Agents: Ontology-Grounded Simulation and Trust Certification
The authors introduce a three-part ontology-based verification system for AI agents that generates regulatory and adversarial test scenarios and issues machine-verifiable trust certificates, with pilot results indicat...
-
SentGuard: Sentence-Level Streaming Guardrails for Large Language Models
SentGuard achieves 90.5% detection of unsafe cases within two sentences at 7.41% false positive rate by operating at sentence boundaries during LLM streaming generation.
-
VERA-MH: Validation of Ethical and Responsible AI in Mental Health
VERA-MH is a clinically-guided three-step evaluation framework for assessing how safely chatbots respond to users expressing suicidal ideation.
-
Evaluating AI-Generated Images of Cultural Artifacts with Community-Informed Rubrics
Community members from the UK blind community, Kerala, and Tamil Nadu helped define what counts as culturally appropriate depictions of artifacts, and the authors tested whether those definitions can be turned into re...
-
Safe for Whom? Rethinking How We Evaluate the Safety of LLMs for Real Users
LLM safety evaluations for personal advice must test responses against diverse user vulnerability profiles, since context-blind ratings overestimate safety and realistic prompt context does not fix the problem.
-
Evolve the Method, Not the Prompts: Evolutionary Synthesis of Jailbreak Attacks on LLMs
EvoSynth evolves code-based jailbreak algorithms via multi-agent self-correction, reaching 85.5% ASR on Claude-Sonnet-4.5 and 95.9% average across targets with greater diversity.
-
A Closer Look at the Existing Risks of Generative AI: Mapping the Who, What, and How of Real-World Incidents
Analysis of 499 generative AI incidents shows use-related failures predominate and frequently harm non-users, producing a distinct risk profile from traditional AI.
-
Anti-Prompt: Image Protection against Text-Guided Image-to-Video Generation
Anti-Prompt adds imperceptible perturbations to images that disrupt text-guided I2V generation by attenuating text-conditioned pathways, achieving protection on two model architectures with a new Video-LLM evaluation ...
-
Solipsistic Superintelligence is Unlikely to be Cooperative
Solipsistic superintelligence developed via unilateral optimization is unlikely to cooperate due to endogenous non-stationarity creating an unclosable train-test-deploy gap.
-
VERA-MH: Validation of Ethical and Responsible AI in Mental Health
VERA-MH is a clinically guided evaluation framework that simulates crisis conversations, judges responses with a yes/no rubric, and rates chatbot safety on suicidal ideation risks.
-
Evaluating AI-Generated Images of Cultural Artifacts with Community-Informed Rubrics
Case studies with blind UK residents and people from Kerala and Tamil Nadu demonstrate that community input at the systematization stage produces culturally grounded definitions of appropriateness for text-to-image mo...
-
Computational Hermeneutics: Evaluating generative AI as a cultural technology
Generative AI should be evaluated through computational hermeneutics using iterative, human-inclusive benchmarks that measure cultural context rather than isolated model outputs.
-
Telling Speculative Stories to Help Humans Imagine the Harms of Healthcare AI
Speculative storytelling and multi-agent discussions enable users to identify a broader range of harms from healthcare AI, with study participants distributing responses more evenly across 17 harm types compared to co...
-
GPT-4o System Card
GPT-4o is OpenAI's end-to-end multimodal model with human-like audio latency, improved non-English text performance, stronger vision and audio understanding, and accompanying safety evaluations.
-
Ceci n'est pas une explication: Evaluating Explanation Failures as Explainability Pitfalls in Language Learning Systems
AI explanations in language learning often fail across six dimensions like diagnostic accuracy and self-regulation support, creating hidden risks that demand better evaluation frameworks such as L2-Bench.
-
Ceci n'est pas une explication: Evaluating Explanation Failures as Explainability Pitfalls in Language Learning Systems
Introduces L2-Bench benchmark for AI feedback in language education across six dimensions and identifies explainability pitfalls in AI-generated explanations that appear helpful but are flawed.
-
What People See (and Miss) About Generative AI Risks: Perceptions of Failures, Risks, and Who Should Address Them
A validated survey instrument grounded in real GenAI incidents reveals public perceptions of failure modes, risks, and stakeholder responsibilities, showing potential for guiding AI literacy efforts.
-
The coordination gap in frontier AI safety policies
Frontier AI safety policies have a structural coordination gap caused by diffuse benefits and concentrated costs, which can be addressed by adapting precommitment and shared response protocols from other high-risk domains.
-
Designing Incident Reporting Systems for Harms from General-Purpose AI
A framework with seven dimensions for AI incident reporting systems is developed from literature and case studies in safety-critical industries to guide institutional design choices.
-
Online Safety Monitoring for LLMs
Simple thresholding on an external verifier signal, calibrated by risk control, performs competitively with sequential hypothesis testing monitors on math reasoning and red-teaming datasets.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.