{"record_type":"pith_number_record","schema_url":"https://pith.science/schemas/pith-number/v1.json","pith_number":"pith:2025:MSLRSR3MTSAHDW7QNMSIX2WXNG","short_pith_number":"pith:MSLRSR3M","schema_version":"1.0","canonical_sha256":"649719476c9c8071dbf06b248bead769bebcbca10bf36aa7b7f30a82a1865720","source":{"kind":"arxiv","id":"2505.23802","version":2},"attestation_state":"computed","paper":{"title":"MedHELM: Holistic Evaluation of Large Language Models for Medical Tasks","license":"http://creativecommons.org/licenses/by/4.0/","headline":"","cross_cats":["cs.AI"],"primary_cat":"cs.CL","authors_text":"Aditya Sharma, Akshay Chaudhari, Akshay Swaminathan, Albert S. Chiou, Alex Alekseyev, Alyssa Unell, Anurang Revri, Asad Aali, Ashwin Nayak, Aydin Zahedivash, Bethel Mieso, Bilal Mawji, Birju Patel, Brian Soetikno, Carlene Lugtu, Chia-Chun Chiang, Christy Hong, Danton Char, Dev Dash, Dong-han Yao, Eduardo Reis, Emily Alsentzer, Eric Horvitz, Ethan Goh, Faraz Ghoddusi, Fateme Nateghi, Francois Grolleau, Hannah Kirsch, Hao Qiu, Harshita Sharma, Hejie Cui, Hinesh Patel, Jarrod Helzer, Jason Alan Fries, Jenelle Jindal, Jennifer Lee, Jonathan Chen, Juan M. Banda, Kameron Black, Keith Morse, Kevin Schulman, Lance Downing, Leonardo Schettini, Matthew P. Lungren, Mehr Kashyap, Mert Oez, Michael F. Gensheimer, Michael Wornow, Miguel Fuentes, Mike Pfeffer, Mohana Roy, Nerissa Ambers, Nigam H. Shah, Nikesh Kotecha, Nima Aghaeepour, Nirmal Ravi, Oluseyi Fayanju, Percy Liang, Philip Chung, Rachna Saralkar, Robson Capasso, Roxana Daneshjou, Sanmi Koyejo, Sergios Gatidis, Shivam Vedak, Shreya Shah, Shrey Jain, Sneha S. Jain, Steven Lin, Suhana Bedi, Thomas Wang, Tho Pham, Timothy Keyes, Tony Lee, Vanessa Kennedy, Vasu Divi, Vicky Zhou, Vikas Kakkar, Wen-wai Yim, Yair Bannett, Yifan Mai","submitted_at":"2025-05-26T22:55:49Z","abstract_excerpt":"While large language models (LLMs) achieve near-perfect scores on medical licensing exams, these evaluations inadequately reflect the complexity and diversity of real-world clinical practice. We introduce MedHELM, an extensible evaluation framework for assessing LLM performance for medical tasks with three key contributions. First, a clinician-validated taxonomy spanning 5 categories, 22 subcategories, and 121 tasks developed with 29 clinicians. Second, a comprehensive benchmark suite comprising 35 benchmarks (17 existing, 18 newly formulated) providing complete coverage of all categories and "},"verification_status":{"content_addressed":true,"pith_receipt":true,"author_attested":false,"weak_author_claims":0,"strong_author_claims":0,"externally_anchored":false,"storage_verified":false,"citation_signatures":0,"replication_records":0,"graph_snapshot":true,"references_resolved":false,"formal_links_present":false},"canonical_record":{"source":{"id":"2505.23802","kind":"arxiv","version":2},"metadata":{"license":"http://creativecommons.org/licenses/by/4.0/","primary_cat":"cs.CL","submitted_at":"2025-05-26T22:55:49Z","cross_cats_sorted":["cs.AI"],"title_canon_sha256":"a19084995afa31ab5ccde2a1e0a840b0f3f15eb6fe01051fa5cc3caaaad2d85e","abstract_canon_sha256":"56d5f3e9e7266a1d5fb605e308e95ae04d1c5f810305abf54c0d0f37e9d5c928"},"schema_version":"1.0"},"receipt":{"kind":"pith_receipt","key_id":"pith-v1-2026-05","algorithm":"ed25519","signed_at":"2026-07-05T11:14:08.832800Z","signature_b64":"M6p9/Yn3MAP8OvxnVhRYeOIEBR695lOH2OXh0azbqfyFGW5hfhSIRdfyNSyNsUJPpTEVVXjDnxZJZEd1bp78Bg==","signed_message":"canonical_sha256_bytes","builder_version":"pith-number-builder-2026-05-17-v1","receipt_version":"0.3","canonical_sha256":"649719476c9c8071dbf06b248bead769bebcbca10bf36aa7b7f30a82a1865720","last_reissued_at":"2026-07-05T11:14:08.832278Z","signature_status":"signed_v1","first_computed_at":"2026-07-05T11:14:08.832278Z","public_key_fingerprint":"8d4b5ee74e4693bcd1df2446408b0d54"},"graph_snapshot":{"paper":{"title":"MedHELM: Holistic Evaluation of Large Language Models for Medical Tasks","license":"http://creativecommons.org/licenses/by/4.0/","headline":"","cross_cats":["cs.AI"],"primary_cat":"cs.CL","authors_text":"Aditya Sharma, Akshay Chaudhari, Akshay Swaminathan, Albert S. Chiou, Alex Alekseyev, Alyssa Unell, Anurang Revri, Asad Aali, Ashwin Nayak, Aydin Zahedivash, Bethel Mieso, Bilal Mawji, Birju Patel, Brian Soetikno, Carlene Lugtu, Chia-Chun Chiang, Christy Hong, Danton Char, Dev Dash, Dong-han Yao, Eduardo Reis, Emily Alsentzer, Eric Horvitz, Ethan Goh, Faraz Ghoddusi, Fateme Nateghi, Francois Grolleau, Hannah Kirsch, Hao Qiu, Harshita Sharma, Hejie Cui, Hinesh Patel, Jarrod Helzer, Jason Alan Fries, Jenelle Jindal, Jennifer Lee, Jonathan Chen, Juan M. Banda, Kameron Black, Keith Morse, Kevin Schulman, Lance Downing, Leonardo Schettini, Matthew P. Lungren, Mehr Kashyap, Mert Oez, Michael F. Gensheimer, Michael Wornow, Miguel Fuentes, Mike Pfeffer, Mohana Roy, Nerissa Ambers, Nigam H. Shah, Nikesh Kotecha, Nima Aghaeepour, Nirmal Ravi, Oluseyi Fayanju, Percy Liang, Philip Chung, Rachna Saralkar, Robson Capasso, Roxana Daneshjou, Sanmi Koyejo, Sergios Gatidis, Shivam Vedak, Shreya Shah, Shrey Jain, Sneha S. Jain, Steven Lin, Suhana Bedi, Thomas Wang, Tho Pham, Timothy Keyes, Tony Lee, Vanessa Kennedy, Vasu Divi, Vicky Zhou, Vikas Kakkar, Wen-wai Yim, Yair Bannett, Yifan Mai","submitted_at":"2025-05-26T22:55:49Z","abstract_excerpt":"While large language models (LLMs) achieve near-perfect scores on medical licensing exams, these evaluations inadequately reflect the complexity and diversity of real-world clinical practice. We introduce MedHELM, an extensible evaluation framework for assessing LLM performance for medical tasks with three key contributions. First, a clinician-validated taxonomy spanning 5 categories, 22 subcategories, and 121 tasks developed with 29 clinicians. Second, a comprehensive benchmark suite comprising 35 benchmarks (17 existing, 18 newly formulated) providing complete coverage of all categories and "},"claims":{"count":0,"items":[],"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"source":{"id":"2505.23802","kind":"arxiv","version":2},"verdict":{"id":null,"model_set":{},"created_at":null,"strongest_claim":"","one_line_summary":"","pipeline_version":null,"weakest_assumption":"","pith_extraction_headline":""},"integrity":{"clean":true,"summary":{"advisory":0,"critical":0,"by_detector":{},"informational":0},"endpoint":"/pith/2505.23802/integrity.json","findings":[],"available":true,"detectors_run":[],"snapshot_sha256":"c28c3603d3b5d939e8dc4c7e95fa8dfce3d595e45f758748cecf8e644a296938"},"references":{"count":0,"sample":[],"resolved_work":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57","internal_anchors":0},"formal_canon":{"evidence_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"author_claims":{"count":0,"strong_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"builder_version":"pith-number-builder-2026-05-17-v1"},"aliases":[{"alias_kind":"arxiv","alias_value":"2505.23802","created_at":"2026-07-05T11:14:08.832341+00:00"},{"alias_kind":"arxiv_version","alias_value":"2505.23802v2","created_at":"2026-07-05T11:14:08.832341+00:00"},{"alias_kind":"doi","alias_value":"10.48550/arxiv.2505.23802","created_at":"2026-07-05T11:14:08.832341+00:00"},{"alias_kind":"pith_short_12","alias_value":"MSLRSR3MTSAH","created_at":"2026-07-05T11:14:08.832341+00:00"},{"alias_kind":"pith_short_16","alias_value":"MSLRSR3MTSAHDW7Q","created_at":"2026-07-05T11:14:08.832341+00:00"},{"alias_kind":"pith_short_8","alias_value":"MSLRSR3M","created_at":"2026-07-05T11:14:08.832341+00:00"}],"events":[],"event_summary":{},"paper_claims":[],"inbound_citations":{"count":20,"internal_anchor_count":0,"sample":[{"citing_arxiv_id":"2607.02175","citing_title":"A rubric-based controlled comparison of frontier language models on expert-authored clinical reasoning tasks","ref_index":1,"is_internal_anchor":false},{"citing_arxiv_id":"2606.07853","citing_title":"Beyond English benchmarks: clinical llm evaluation in Brazilian Portuguese","ref_index":39,"is_internal_anchor":false},{"citing_arxiv_id":"2606.01961","citing_title":"AutoMedBench: Towards Medical AutoResearch with Agentic AI Models","ref_index":6,"is_internal_anchor":false},{"citing_arxiv_id":"2606.31608","citing_title":"CLExEval: A Human-in-the-Loop Framework for Qualitative Evaluation of LLM Clinical Reasoning","ref_index":53,"is_internal_anchor":false},{"citing_arxiv_id":"2605.16679","citing_title":"CHI-Bench: Can AI Agents Automate End-to-End, Long-Horizon, Policy-Rich Healthcare Workflows?","ref_index":8,"is_internal_anchor":false},{"citing_arxiv_id":"2605.17765","citing_title":"AURORA: Contextual Orthogonalization for Geometric Representation Learning in Healthcare Foundation Models","ref_index":35,"is_internal_anchor":false},{"citing_arxiv_id":"2512.20983","citing_title":"Automatic Replication of LLM Mistakes in Medical Conversations","ref_index":3,"is_internal_anchor":false},{"citing_arxiv_id":"2512.19691","citing_title":"Scalable Stewardship of an LLM-Assisted Clinical Benchmark with Physician Oversight","ref_index":15,"is_internal_anchor":false},{"citing_arxiv_id":"2604.02359","citing_title":"Using LLM-as-a-Judge/Jury to Advance Scalable, Clinically-Validated Safety Evaluations of Model Responses to Users Demonstrating Psychosis","ref_index":9,"is_internal_anchor":false},{"citing_arxiv_id":"2605.11206","citing_title":"Instructions Shape Production of Language, not Processing","ref_index":121,"is_internal_anchor":false},{"citing_arxiv_id":"2605.11206","citing_title":"Instructions Shape Production of Language, not Processing","ref_index":121,"is_internal_anchor":false},{"citing_arxiv_id":"2605.08685","citing_title":"Event Fields: Learning Latent Event Structure for Waveform Foundation Models","ref_index":31,"is_internal_anchor":false},{"citing_arxiv_id":"2605.08445","citing_title":"Measuring What Matters: Benchmarking Generative, Multimodal, and Agentic AI in Healthcare","ref_index":1,"is_internal_anchor":false},{"citing_arxiv_id":"2605.09675","citing_title":"CodeClinic: Evaluating Automation of Coding Skills for Clinical Reasoning Agents","ref_index":6,"is_internal_anchor":false},{"citing_arxiv_id":"2605.09765","citing_title":"WISTERIA: Learning Clinical Representations from Noisy Supervision via Multi-View Consistency in Electronic Health Records","ref_index":33,"is_internal_anchor":false},{"citing_arxiv_id":"2605.04012","citing_title":"SymptomAI: Toward a Conversational AI Agent for Everyday Symptom Assessment","ref_index":2,"is_internal_anchor":false},{"citing_arxiv_id":"2605.09584","citing_title":"CLR-voyance: Reinforcing Open-Ended Reasoning for Inpatient Clinical Decision Support with Outcome-Aware Rubrics","ref_index":22,"is_internal_anchor":false},{"citing_arxiv_id":"2605.04012","citing_title":"SymptomAI: Toward a Conversational AI Agent for Everyday Symptom Assessment","ref_index":2,"is_internal_anchor":false},{"citing_arxiv_id":"2604.24700","citing_title":"Green Shielding: A User-Centric Approach Towards Trustworthy AI","ref_index":62,"is_internal_anchor":false},{"citing_arxiv_id":"2604.11133","citing_title":"How Robust Are Large Language Models for Clinical Numeracy? An Empirical Study on Numerical Reasoning Abilities in Clinical Contexts","ref_index":1,"is_internal_anchor":false}]},"formal_canon":{"evidence_count":0,"sample":[],"anchors":[]},"links":{"html":"https://pith.science/pith/MSLRSR3MTSAHDW7QNMSIX2WXNG","json":"https://pith.science/pith/MSLRSR3MTSAHDW7QNMSIX2WXNG.json","graph_json":"https://pith.science/api/pith-number/MSLRSR3MTSAHDW7QNMSIX2WXNG/graph.json","events_json":"https://pith.science/api/pith-number/MSLRSR3MTSAHDW7QNMSIX2WXNG/events.json","paper":"https://pith.science/paper/MSLRSR3M"},"agent_actions":{"view_html":"https://pith.science/pith/MSLRSR3MTSAHDW7QNMSIX2WXNG","download_json":"https://pith.science/pith/MSLRSR3MTSAHDW7QNMSIX2WXNG.json","view_paper":"https://pith.science/paper/MSLRSR3M","resolve_alias":"https://pith.science/api/pith-number/resolve?arxiv=2505.23802&json=true","fetch_graph":"https://pith.science/api/pith-number/MSLRSR3MTSAHDW7QNMSIX2WXNG/graph.json","fetch_events":"https://pith.science/api/pith-number/MSLRSR3MTSAHDW7QNMSIX2WXNG/events.json","actions":{"anchor_timestamp":"https://pith.science/pith/MSLRSR3MTSAHDW7QNMSIX2WXNG/action/timestamp_anchor","attest_storage":"https://pith.science/pith/MSLRSR3MTSAHDW7QNMSIX2WXNG/action/storage_attestation","attest_author":"https://pith.science/pith/MSLRSR3MTSAHDW7QNMSIX2WXNG/action/author_attestation","sign_citation":"https://pith.science/pith/MSLRSR3MTSAHDW7QNMSIX2WXNG/action/citation_signature","submit_replication":"https://pith.science/pith/MSLRSR3MTSAHDW7QNMSIX2WXNG/action/replication_record"}},"created_at":"2026-07-05T11:14:08.832341+00:00","updated_at":"2026-07-05T11:14:08.832341+00:00"}