{"record_type":"pith_number_record","schema_url":"https://pith.science/schemas/pith-number/v1.json","pith_number":"pith:2025:CW2YDRVT6XF6AY3LM2FDF2ISOA","short_pith_number":"pith:CW2YDRVT","schema_version":"1.0","canonical_sha256":"15b581c6b3f5cbe0636b668a32e912703972121462d54dfcbe31eb21f0a53f60","source":{"kind":"arxiv","id":"2503.05731","version":2},"attestation_state":"computed","paper":{"title":"AILuminate: Introducing v1.0 of the AI Risk and Reliability Benchmark from MLCommons","license":"http://creativecommons.org/licenses/by/4.0/","headline":"","cross_cats":["cs.AI"],"primary_cat":"cs.CY","authors_text":"Aakash Gupta, Abhinav Singh, Abolfazl Shahbazi, Adina Williams, Agasthya Gangavarapu, Ahmed Ahmed, Alicia Parrish, Amy Russ, Ananya Gangavarapu, Ben Huang, Benjamin Rukundo, Bertie Vidgen, Bhaktipriya Radharapu, Canyu Chen, Carlos Mougn, Chenhe Gu, Chris Knotz, Cigdem Patlak, Claire V. Hardgrove, Cody Coleman, Debojyoti Dutta, Declan Grabb, Eileen Long, Eleonora Presani, Elie Alhajjar, Emil Joswin, Fazl Barez, Fedor Zhdanov, Felix Friedrich, Foutse Khomh, Heather Frase, Hussain Chinoy, James Ezick, James Gealy, James Goel, Jibin Rajan Varghese, Joachim Baumann, Joaquin Vanschoren, Jonathan Bennion, Joseph Marvin Imperial, Julien Delaunay, Kashyap Ramanandula Manjusha, Kelvin N. Manyeki, Kenneth Fricklas, Kevin Lu, Kongtao Chen, Kurt Bollacker, Luis Oala, Mala Kumar, Malek Ben Salem, Manil Maskey, Marisa Ferrara Boston, Matt Lavrisa, Max Bartolo, Mike Kuniavsky, Mubashara Akhtar, Murali Emani, Natan Vidra, Nicolas Miailhe, Nina Vasan, Patrick Schramowski, Paul R\\\"ottger, Percy Liang, Peter Mattson, Pierre Peign\\'e, Priyanka Mary Mammen, Quentin Feuillade--Montixi, Rafael Gold, Rajat Ghosh, Rajat Sahay, Rajat Shinde, Rebecca Weiss, Robert Sullivan, Roman Eng, Ryan Tsang, Sandeep Madireddy, Sarah Luger, Sasidhar Kunapuli, Satyapriya Krishna, Saurav Sahay, Sean McGregor, Seshakrishna Jitendar, Shachi H Kumar, Shaona Ghosh, Sree Bhargavi Balija, Stephen Singam, Sujata Goswami, Supheakmungkol Sarin, Tarunima Prabhakar, Tianhao Li, Tom David, Tuesday, Usman Gohar, Vassil Tashev, Venelin Kovatchev, Vincent Alessi, Virendra Mehta, Vithursan Thangarasa, Wiebke Hutiri, Xianjun Yang, Xudong Han, Yifan Mai","submitted_at":"2025-02-19T05:58:52Z","abstract_excerpt":"The rapid advancement and deployment of AI systems have created an urgent need for standard safety-evaluation frameworks. This paper introduces AILuminate v1.0, the first comprehensive industry-standard benchmark for assessing AI-product risk and reliability. Its development employed an open process that included participants from multiple fields. The benchmark evaluates an AI system's resistance to prompts designed to elicit dangerous, illegal, or undesirable behavior in 12 hazard categories, including violent crimes, nonviolent crimes, sex-related crimes, child sexual exploitation, indiscrim"},"verification_status":{"content_addressed":true,"pith_receipt":true,"author_attested":false,"weak_author_claims":0,"strong_author_claims":0,"externally_anchored":false,"storage_verified":false,"citation_signatures":0,"replication_records":0,"graph_snapshot":true,"references_resolved":false,"formal_links_present":false},"canonical_record":{"source":{"id":"2503.05731","kind":"arxiv","version":2},"metadata":{"license":"http://creativecommons.org/licenses/by/4.0/","primary_cat":"cs.CY","submitted_at":"2025-02-19T05:58:52Z","cross_cats_sorted":["cs.AI"],"title_canon_sha256":"cc16ce2ba0bbfed632a234dd1b35b8965fcd62e13a2d99db8988666c678ef356","abstract_canon_sha256":"0010f36625a9bb5c13d3628d695616438e5564af9c35b55ce5595eae34b2ea58"},"schema_version":"1.0"},"receipt":{"kind":"pith_receipt","key_id":"pith-v1-2026-05","algorithm":"ed25519","signed_at":"2026-07-05T10:51:05.874075Z","signature_b64":"ozElNZbw1LXTXZ4mfm3qNly0XJEvISkDcRbZ2sBlz0JY0bgu6KPDxk5lqLGVbOkqqOgtnT+3QCPaiYxRhJqDAw==","signed_message":"canonical_sha256_bytes","builder_version":"pith-number-builder-2026-05-17-v1","receipt_version":"0.3","canonical_sha256":"15b581c6b3f5cbe0636b668a32e912703972121462d54dfcbe31eb21f0a53f60","last_reissued_at":"2026-07-05T10:51:05.873556Z","signature_status":"signed_v1","first_computed_at":"2026-07-05T10:51:05.873556Z","public_key_fingerprint":"8d4b5ee74e4693bcd1df2446408b0d54"},"graph_snapshot":{"paper":{"title":"AILuminate: Introducing v1.0 of the AI Risk and Reliability Benchmark from MLCommons","license":"http://creativecommons.org/licenses/by/4.0/","headline":"","cross_cats":["cs.AI"],"primary_cat":"cs.CY","authors_text":"Aakash Gupta, Abhinav Singh, Abolfazl Shahbazi, Adina Williams, Agasthya Gangavarapu, Ahmed Ahmed, Alicia Parrish, Amy Russ, Ananya Gangavarapu, Ben Huang, Benjamin Rukundo, Bertie Vidgen, Bhaktipriya Radharapu, Canyu Chen, Carlos Mougn, Chenhe Gu, Chris Knotz, Cigdem Patlak, Claire V. Hardgrove, Cody Coleman, Debojyoti Dutta, Declan Grabb, Eileen Long, Eleonora Presani, Elie Alhajjar, Emil Joswin, Fazl Barez, Fedor Zhdanov, Felix Friedrich, Foutse Khomh, Heather Frase, Hussain Chinoy, James Ezick, James Gealy, James Goel, Jibin Rajan Varghese, Joachim Baumann, Joaquin Vanschoren, Jonathan Bennion, Joseph Marvin Imperial, Julien Delaunay, Kashyap Ramanandula Manjusha, Kelvin N. Manyeki, Kenneth Fricklas, Kevin Lu, Kongtao Chen, Kurt Bollacker, Luis Oala, Mala Kumar, Malek Ben Salem, Manil Maskey, Marisa Ferrara Boston, Matt Lavrisa, Max Bartolo, Mike Kuniavsky, Mubashara Akhtar, Murali Emani, Natan Vidra, Nicolas Miailhe, Nina Vasan, Patrick Schramowski, Paul R\\\"ottger, Percy Liang, Peter Mattson, Pierre Peign\\'e, Priyanka Mary Mammen, Quentin Feuillade--Montixi, Rafael Gold, Rajat Ghosh, Rajat Sahay, Rajat Shinde, Rebecca Weiss, Robert Sullivan, Roman Eng, Ryan Tsang, Sandeep Madireddy, Sarah Luger, Sasidhar Kunapuli, Satyapriya Krishna, Saurav Sahay, Sean McGregor, Seshakrishna Jitendar, Shachi H Kumar, Shaona Ghosh, Sree Bhargavi Balija, Stephen Singam, Sujata Goswami, Supheakmungkol Sarin, Tarunima Prabhakar, Tianhao Li, Tom David, Tuesday, Usman Gohar, Vassil Tashev, Venelin Kovatchev, Vincent Alessi, Virendra Mehta, Vithursan Thangarasa, Wiebke Hutiri, Xianjun Yang, Xudong Han, Yifan Mai","submitted_at":"2025-02-19T05:58:52Z","abstract_excerpt":"The rapid advancement and deployment of AI systems have created an urgent need for standard safety-evaluation frameworks. This paper introduces AILuminate v1.0, the first comprehensive industry-standard benchmark for assessing AI-product risk and reliability. Its development employed an open process that included participants from multiple fields. The benchmark evaluates an AI system's resistance to prompts designed to elicit dangerous, illegal, or undesirable behavior in 12 hazard categories, including violent crimes, nonviolent crimes, sex-related crimes, child sexual exploitation, indiscrim"},"claims":{"count":0,"items":[],"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"source":{"id":"2503.05731","kind":"arxiv","version":2},"verdict":{"id":null,"model_set":{},"created_at":null,"strongest_claim":"","one_line_summary":"","pipeline_version":null,"weakest_assumption":"","pith_extraction_headline":""},"integrity":{"clean":true,"summary":{"advisory":0,"critical":0,"by_detector":{},"informational":0},"endpoint":"/pith/2503.05731/integrity.json","findings":[],"available":true,"detectors_run":[],"snapshot_sha256":"c28c3603d3b5d939e8dc4c7e95fa8dfce3d595e45f758748cecf8e644a296938"},"references":{"count":0,"sample":[],"resolved_work":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57","internal_anchors":0},"formal_canon":{"evidence_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"author_claims":{"count":0,"strong_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"builder_version":"pith-number-builder-2026-05-17-v1"},"aliases":[{"alias_kind":"arxiv","alias_value":"2503.05731","created_at":"2026-07-05T10:51:05.873616+00:00"},{"alias_kind":"arxiv_version","alias_value":"2503.05731v2","created_at":"2026-07-05T10:51:05.873616+00:00"},{"alias_kind":"doi","alias_value":"10.48550/arxiv.2503.05731","created_at":"2026-07-05T10:51:05.873616+00:00"},{"alias_kind":"pith_short_12","alias_value":"CW2YDRVT6XF6","created_at":"2026-07-05T10:51:05.873616+00:00"},{"alias_kind":"pith_short_16","alias_value":"CW2YDRVT6XF6AY3L","created_at":"2026-07-05T10:51:05.873616+00:00"},{"alias_kind":"pith_short_8","alias_value":"CW2YDRVT","created_at":"2026-07-05T10:51:05.873616+00:00"}],"events":[],"event_summary":{},"paper_claims":[],"inbound_citations":{"count":18,"internal_anchor_count":0,"sample":[{"citing_arxiv_id":"2606.19887","citing_title":"FinRED: An Expert-Guided Benchmark Generation and Evaluation Framework for Financial LLM Red-Teaming","ref_index":9,"is_internal_anchor":false},{"citing_arxiv_id":"2606.09809","citing_title":"Evaluation Cards: An Interpretive Layer for AI Evaluation Reporting","ref_index":45,"is_internal_anchor":false},{"citing_arxiv_id":"2605.31381","citing_title":"LLM Judges Inconsistently Disagree Across Safety Criteria and Harm Categories","ref_index":1,"is_internal_anchor":false},{"citing_arxiv_id":"2606.29887","citing_title":"SafePyramid: A Hierarchical Benchmark for In-context Policy Guardrailing","ref_index":15,"is_internal_anchor":false},{"citing_arxiv_id":"2605.28137","citing_title":"No Safe Dose: How Training Data Drives Unsafe Image Generation","ref_index":21,"is_internal_anchor":false},{"citing_arxiv_id":"2606.00359","citing_title":"Next-Billion AI Index: The compass for AI utility and adoption in the global majority","ref_index":94,"is_internal_anchor":false},{"citing_arxiv_id":"2606.09408","citing_title":"Can Data Work be Reparative?","ref_index":23,"is_internal_anchor":false},{"citing_arxiv_id":"2605.22643","citing_title":"Boiling the Frog: A Multi-Turn Benchmark for Agentic Safety","ref_index":36,"is_internal_anchor":false},{"citing_arxiv_id":"2605.22643","citing_title":"Boiling the Frog: A Multi-Turn Benchmark for Agentic Safety","ref_index":36,"is_internal_anchor":false},{"citing_arxiv_id":"2605.20203","citing_title":"GrandGuard: Taxonomy, Benchmark, and Safeguards for Elderly-Chatbot Interaction Safety","ref_index":3,"is_internal_anchor":false},{"citing_arxiv_id":"2605.16471","citing_title":"From AI-Generated Content to Agentic Action: Security and Safety Threats in Generative AI","ref_index":45,"is_internal_anchor":false},{"citing_arxiv_id":"2601.06033","citing_title":"How Generative AI Empowers Attackers and Defenders Across the Trust & Safety Landscape","ref_index":34,"is_internal_anchor":false},{"citing_arxiv_id":"2605.10442","citing_title":"StereoTales: A Multilingual Framework for Open-Ended Stereotype Discovery in LLMs","ref_index":43,"is_internal_anchor":false},{"citing_arxiv_id":"2605.10442","citing_title":"StereoTales: A Multilingual Framework for Open-Ended Stereotype Discovery in LLMs","ref_index":43,"is_internal_anchor":false},{"citing_arxiv_id":"2605.06213","citing_title":"Beyond Fixed Benchmarks and Worst-Case Attacks: Dynamic Boundary Evaluation for Language Models","ref_index":4,"is_internal_anchor":false},{"citing_arxiv_id":"2604.18487","citing_title":"Adversarial Humanities Benchmark: Results on Stylistic Robustness in Frontier Model Safety","ref_index":44,"is_internal_anchor":false},{"citing_arxiv_id":"2605.01965","citing_title":"Retrieval with Multiple Query Vectors through Anomalous Pattern Detection","ref_index":51,"is_internal_anchor":false},{"citing_arxiv_id":"2605.06652","citing_title":"When No Benchmark Exists: Validating Comparative LLM Safety Scoring Without Ground-Truth Labels","ref_index":21,"is_internal_anchor":false}]},"formal_canon":{"evidence_count":0,"sample":[],"anchors":[]},"links":{"html":"https://pith.science/pith/CW2YDRVT6XF6AY3LM2FDF2ISOA","json":"https://pith.science/pith/CW2YDRVT6XF6AY3LM2FDF2ISOA.json","graph_json":"https://pith.science/api/pith-number/CW2YDRVT6XF6AY3LM2FDF2ISOA/graph.json","events_json":"https://pith.science/api/pith-number/CW2YDRVT6XF6AY3LM2FDF2ISOA/events.json","paper":"https://pith.science/paper/CW2YDRVT"},"agent_actions":{"view_html":"https://pith.science/pith/CW2YDRVT6XF6AY3LM2FDF2ISOA","download_json":"https://pith.science/pith/CW2YDRVT6XF6AY3LM2FDF2ISOA.json","view_paper":"https://pith.science/paper/CW2YDRVT","resolve_alias":"https://pith.science/api/pith-number/resolve?arxiv=2503.05731&json=true","fetch_graph":"https://pith.science/api/pith-number/CW2YDRVT6XF6AY3LM2FDF2ISOA/graph.json","fetch_events":"https://pith.science/api/pith-number/CW2YDRVT6XF6AY3LM2FDF2ISOA/events.json","actions":{"anchor_timestamp":"https://pith.science/pith/CW2YDRVT6XF6AY3LM2FDF2ISOA/action/timestamp_anchor","attest_storage":"https://pith.science/pith/CW2YDRVT6XF6AY3LM2FDF2ISOA/action/storage_attestation","attest_author":"https://pith.science/pith/CW2YDRVT6XF6AY3LM2FDF2ISOA/action/author_attestation","sign_citation":"https://pith.science/pith/CW2YDRVT6XF6AY3LM2FDF2ISOA/action/citation_signature","submit_replication":"https://pith.science/pith/CW2YDRVT6XF6AY3LM2FDF2ISOA/action/replication_record"}},"created_at":"2026-07-05T10:51:05.873616+00:00","updated_at":"2026-07-05T10:51:05.873616+00:00"}