{"record_type":"pith_number_record","schema_url":"https://pith.science/schemas/pith-number/v1.json","pith_number":"pith:2024:6IL5SNEOH3WB3JUDON72G4ZQOV","short_pith_number":"pith:6IL5SNEO","schema_version":"1.0","canonical_sha256":"f217d9348e3eec1da683737fa37330757253dc2301ef01507740739934d801c8","source":{"kind":"arxiv","id":"2406.14598","version":2},"attestation_state":"computed","paper":{"title":"SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusal","license":"http://arxiv.org/licenses/nonexclusive-distrib/1.0/","headline":"","cross_cats":[],"primary_cat":"cs.AI","authors_text":"Bo Li, Boyi Wei, Dacheng Li, Danqi Chen, Kai Li, Kaixuan Huang, Luxi He, Peter Henderson, Prateek Mittal, Ruoxi Jia, Tinghao Xie, Udari Madhushani Sehwag, Xiangyu Qi, Yangsibo Huang, Ying Sheng, Yi Zeng","submitted_at":"2024-06-20T17:56:07Z","abstract_excerpt":"Evaluating aligned large language models' (LLMs) ability to recognize and reject unsafe user requests is crucial for safe, policy-compliant deployments. Existing evaluation efforts, however, face three limitations that we address with SORRY-Bench, our proposed benchmark. First, existing methods often use coarse-grained taxonomies of unsafe topics, and are over-representing some fine-grained topics. For example, among the ten existing datasets that we evaluated, tests for refusals of self-harm instructions are over 3x less represented than tests for fraudulent activities. SORRY-Bench improves o"},"verification_status":{"content_addressed":true,"pith_receipt":true,"author_attested":false,"weak_author_claims":0,"strong_author_claims":0,"externally_anchored":false,"storage_verified":false,"citation_signatures":0,"replication_records":0,"graph_snapshot":true,"references_resolved":false,"formal_links_present":false},"canonical_record":{"source":{"id":"2406.14598","kind":"arxiv","version":2},"metadata":{"license":"http://arxiv.org/licenses/nonexclusive-distrib/1.0/","primary_cat":"cs.AI","submitted_at":"2024-06-20T17:56:07Z","cross_cats_sorted":[],"title_canon_sha256":"c2e73424dbc4f04619d46ab8005fedbe3e954404747798b3d1ad88a7d4b14513","abstract_canon_sha256":"7d4e293df6031a47a7b625cc0f99eae6537624dfcb1de49947fa06a3f5cf8fa4"},"schema_version":"1.0"},"receipt":{"kind":"pith_receipt","key_id":"pith-v1-2026-05","algorithm":"ed25519","signed_at":"2026-07-05T10:22:01.456762Z","signature_b64":"EoJPkuWh9NjRCV7EU8s2lMiwR18c9xmNMxew2uSeXlTJtfw9qhbzM602ouK5lSESvksGWP6PtBKkykq1PlcECw==","signed_message":"canonical_sha256_bytes","builder_version":"pith-number-builder-2026-05-17-v1","receipt_version":"0.3","canonical_sha256":"f217d9348e3eec1da683737fa37330757253dc2301ef01507740739934d801c8","last_reissued_at":"2026-07-05T10:22:01.456080Z","signature_status":"signed_v1","first_computed_at":"2026-07-05T10:22:01.456080Z","public_key_fingerprint":"8d4b5ee74e4693bcd1df2446408b0d54"},"graph_snapshot":{"paper":{"title":"SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusal","license":"http://arxiv.org/licenses/nonexclusive-distrib/1.0/","headline":"","cross_cats":[],"primary_cat":"cs.AI","authors_text":"Bo Li, Boyi Wei, Dacheng Li, Danqi Chen, Kai Li, Kaixuan Huang, Luxi He, Peter Henderson, Prateek Mittal, Ruoxi Jia, Tinghao Xie, Udari Madhushani Sehwag, Xiangyu Qi, Yangsibo Huang, Ying Sheng, Yi Zeng","submitted_at":"2024-06-20T17:56:07Z","abstract_excerpt":"Evaluating aligned large language models' (LLMs) ability to recognize and reject unsafe user requests is crucial for safe, policy-compliant deployments. Existing evaluation efforts, however, face three limitations that we address with SORRY-Bench, our proposed benchmark. First, existing methods often use coarse-grained taxonomies of unsafe topics, and are over-representing some fine-grained topics. For example, among the ten existing datasets that we evaluated, tests for refusals of self-harm instructions are over 3x less represented than tests for fraudulent activities. SORRY-Bench improves o"},"claims":{"count":0,"items":[],"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"source":{"id":"2406.14598","kind":"arxiv","version":2},"verdict":{"id":null,"model_set":{},"created_at":null,"strongest_claim":"","one_line_summary":"","pipeline_version":null,"weakest_assumption":"","pith_extraction_headline":""},"integrity":{"clean":true,"summary":{"advisory":0,"critical":0,"by_detector":{},"informational":0},"endpoint":"/pith/2406.14598/integrity.json","findings":[],"available":true,"detectors_run":[],"snapshot_sha256":"c28c3603d3b5d939e8dc4c7e95fa8dfce3d595e45f758748cecf8e644a296938"},"references":{"count":0,"sample":[],"resolved_work":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57","internal_anchors":0},"formal_canon":{"evidence_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"author_claims":{"count":0,"strong_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"builder_version":"pith-number-builder-2026-05-17-v1"},"aliases":[{"alias_kind":"arxiv","alias_value":"2406.14598","created_at":"2026-07-05T10:22:01.456167+00:00"},{"alias_kind":"arxiv_version","alias_value":"2406.14598v2","created_at":"2026-07-05T10:22:01.456167+00:00"},{"alias_kind":"doi","alias_value":"10.48550/arxiv.2406.14598","created_at":"2026-07-05T10:22:01.456167+00:00"},{"alias_kind":"pith_short_12","alias_value":"6IL5SNEOH3WB","created_at":"2026-07-05T10:22:01.456167+00:00"},{"alias_kind":"pith_short_16","alias_value":"6IL5SNEOH3WB3JUD","created_at":"2026-07-05T10:22:01.456167+00:00"},{"alias_kind":"pith_short_8","alias_value":"6IL5SNEO","created_at":"2026-07-05T10:22:01.456167+00:00"}],"events":[],"event_summary":{},"paper_claims":[],"inbound_citations":{"count":28,"internal_anchor_count":0,"sample":[{"citing_arxiv_id":"2606.25782","citing_title":"Do Encoders Suffice? A Systematic Comparison of Encoder and Decoder Safety Judges for LLM Adversarial Evaluation","ref_index":16,"is_internal_anchor":false},{"citing_arxiv_id":"2606.24388","citing_title":"PHANTOM: A Large-Scale Dataset of Multimodal Adversarial Attacks for Vision-Language Models","ref_index":11,"is_internal_anchor":false},{"citing_arxiv_id":"2606.20508","citing_title":"What Do Safety-Aligned LLMs Learn From Mixed Compliance Demonstrations?","ref_index":10,"is_internal_anchor":false},{"citing_arxiv_id":"2606.11316","citing_title":"Sch\\\"utzen: Evaluating LLM Safety in Bulgarian and German Contexts","ref_index":9,"is_internal_anchor":false},{"citing_arxiv_id":"2606.09165","citing_title":"Reliable to Expressive: A Curriculum for Rubric-Following Safety Judges","ref_index":9,"is_internal_anchor":false},{"citing_arxiv_id":"2606.20626","citing_title":"Efficient Safety Benchmarking via Item Response Theory","ref_index":16,"is_internal_anchor":false},{"citing_arxiv_id":"2606.31748","citing_title":"Addressing Over-Refusal in LLMs with Competing Rewards","ref_index":34,"is_internal_anchor":false},{"citing_arxiv_id":"2606.28733","citing_title":"Agentic Abstention: Do Agents Know When to Stop Instead of Act?","ref_index":61,"is_internal_anchor":false},{"citing_arxiv_id":"2605.29659","citing_title":"Opir: Efficient Multi-Task Safety Classification for Toxicity, Jailbreaks, Hate Speech, and Harmful Content","ref_index":39,"is_internal_anchor":false},{"citing_arxiv_id":"2606.21296","citing_title":"Discriminatory Compliance: How LLMs Answer Queries from Protected Groups","ref_index":106,"is_internal_anchor":false},{"citing_arxiv_id":"2411.15594","citing_title":"A Survey on LLM-as-a-Judge","ref_index":178,"is_internal_anchor":false},{"citing_arxiv_id":"2605.21545","citing_title":"RefusalBench: Why Refusal Rate Misranks Frontier LLMs on Biological Research Prompts","ref_index":28,"is_internal_anchor":false},{"citing_arxiv_id":"2509.09708","citing_title":"Beyond I'm Sorry, I Can't: Dissecting Large Language Model Refusal","ref_index":37,"is_internal_anchor":false},{"citing_arxiv_id":"2605.12726","citing_title":"Before the Last Token: Diagnosing Final-Token Safety Probe Failures","ref_index":13,"is_internal_anchor":false},{"citing_arxiv_id":"2605.13334","citing_title":"LLM-Based Persuasion Enables Guardrail Override in Frontier LLMs","ref_index":2,"is_internal_anchor":false},{"citing_arxiv_id":"2604.06233","citing_title":"Blind Refusal: Language Models Refuse to Help Users Evade Unjust, Absurd, and Illegitimate Rules","ref_index":36,"is_internal_anchor":false},{"citing_arxiv_id":"2605.10998","citing_title":"Few-Shot Truly Benign DPO Attack for Jailbreaking LLMs","ref_index":44,"is_internal_anchor":false},{"citing_arxiv_id":"2410.02644","citing_title":"Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents","ref_index":148,"is_internal_anchor":false},{"citing_arxiv_id":"2605.03095","citing_title":"Revisiting JBShield: Breaking and Rebuilding Representation-Level Jailbreak Defenses","ref_index":50,"is_internal_anchor":false},{"citing_arxiv_id":"2412.05579","citing_title":"LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods","ref_index":258,"is_internal_anchor":false},{"citing_arxiv_id":"2605.00706","citing_title":"FinSafetyBench: Evaluating LLM Safety in Real-World Financial Scenarios","ref_index":6,"is_internal_anchor":false},{"citing_arxiv_id":"2605.01123","citing_title":"PERSA: Reinforcement Learning for Professor-Style Personalized Feedback with LLMs","ref_index":19,"is_internal_anchor":false},{"citing_arxiv_id":"2604.18946","citing_title":"Reasoning Structure Matters for Safety Alignment of Reasoning Models","ref_index":9,"is_internal_anchor":false},{"citing_arxiv_id":"2605.01687","citing_title":"MultiBreak: A Scalable and Diverse Multi-turn Jailbreak Benchmark for Evaluating LLM Safety","ref_index":17,"is_internal_anchor":false},{"citing_arxiv_id":"2604.07709","citing_title":"IatroBench: Pre-Registered Evidence of Iatrogenic Harm from AI Safety Measures","ref_index":37,"is_internal_anchor":false}]},"formal_canon":{"evidence_count":0,"sample":[],"anchors":[]},"links":{"html":"https://pith.science/pith/6IL5SNEOH3WB3JUDON72G4ZQOV","json":"https://pith.science/pith/6IL5SNEOH3WB3JUDON72G4ZQOV.json","graph_json":"https://pith.science/api/pith-number/6IL5SNEOH3WB3JUDON72G4ZQOV/graph.json","events_json":"https://pith.science/api/pith-number/6IL5SNEOH3WB3JUDON72G4ZQOV/events.json","paper":"https://pith.science/paper/6IL5SNEO"},"agent_actions":{"view_html":"https://pith.science/pith/6IL5SNEOH3WB3JUDON72G4ZQOV","download_json":"https://pith.science/pith/6IL5SNEOH3WB3JUDON72G4ZQOV.json","view_paper":"https://pith.science/paper/6IL5SNEO","resolve_alias":"https://pith.science/api/pith-number/resolve?arxiv=2406.14598&json=true","fetch_graph":"https://pith.science/api/pith-number/6IL5SNEOH3WB3JUDON72G4ZQOV/graph.json","fetch_events":"https://pith.science/api/pith-number/6IL5SNEOH3WB3JUDON72G4ZQOV/events.json","actions":{"anchor_timestamp":"https://pith.science/pith/6IL5SNEOH3WB3JUDON72G4ZQOV/action/timestamp_anchor","attest_storage":"https://pith.science/pith/6IL5SNEOH3WB3JUDON72G4ZQOV/action/storage_attestation","attest_author":"https://pith.science/pith/6IL5SNEOH3WB3JUDON72G4ZQOV/action/author_attestation","sign_citation":"https://pith.science/pith/6IL5SNEOH3WB3JUDON72G4ZQOV/action/citation_signature","submit_replication":"https://pith.science/pith/6IL5SNEOH3WB3JUDON72G4ZQOV/action/replication_record"}},"created_at":"2026-07-05T10:22:01.456167+00:00","updated_at":"2026-07-05T10:22:01.456167+00:00"}