{"paper":{"title":"A Toolbox for Surfacing Health Equity Harms and Biases in Large Language Models","license":"http://arxiv.org/licenses/nonexclusive-distrib/1.0/","headline":"","cross_cats":["cs.CL","cs.LG"],"primary_cat":"cs.CY","authors_text":"Akeiylah Dewitt, Alan Karthikesalingam, Alanna Walton, Alicia Parrish, Awa Dieng, Chirag Nagpal, Christopher Semturs, Darlene Neal, Greg Corrado, Heather Cole-Lewis, Ivor Horn, Jamila Smith-Loud, Joelle Barral, Karan Singhal, Katherine Heller, Leo Anthony Celi, Liam G. McCoy, Mercy Asiedu, Mike Schaekermann, Negar Rostamzadeh, Nenad Tomasev, Philip Mansfield, Preeti Singh, Qazi Mamunur Rashid, Rory Sayres, Shekoofeh Azizi, Stephen R. Pfohl, Sushant Prakash, Yossi Matias, Yun Liu","submitted_at":"2024-03-18T17:56:37Z","abstract_excerpt":"Large language models (LLMs) hold promise to serve complex health information needs but also have the potential to introduce harm and exacerbate health disparities. Reliably evaluating equity-related model failures is a critical step toward developing systems that promote health equity. We present resources and methodologies for surfacing biases with potential to precipitate equity-related harms in long-form, LLM-generated answers to medical questions and conduct a large-scale empirical case study with the Med-PaLM 2 LLM. Our contributions include a multifactorial framework for human assessmen"},"claims":{"count":0,"items":[],"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"source":{"id":"2403.12025","kind":"arxiv","version":2},"verdict":{"id":null,"model_set":{},"created_at":null,"strongest_claim":"","one_line_summary":"","pipeline_version":null,"weakest_assumption":"","pith_extraction_headline":""},"integrity":{"clean":true,"summary":{"advisory":0,"critical":0,"by_detector":{},"informational":0},"endpoint":"/pith/2403.12025/integrity.json","findings":[],"available":true,"detectors_run":[],"snapshot_sha256":"c28c3603d3b5d939e8dc4c7e95fa8dfce3d595e45f758748cecf8e644a296938"},"references":{"count":0,"sample":[],"resolved_work":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57","internal_anchors":0},"formal_canon":{"evidence_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"author_claims":{"count":0,"strong_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"builder_version":"pith-number-builder-2026-05-17-v1"}