Small open-weight language models can self-optimize prompts for clinical named entity recognition in dental notes, reaching micro F1 of 0.864 after DPO on Qwen2.5-14B.
Beyond the Leaderboard: Rethinking Medical Benchmarks for Large Language Models
2 Pith papers cite this work. Polarity classification is still indexing.
2
Pith papers citing it
abstract
Large language models (LLMs) show significant potential in healthcare, prompting numerous benchmarks to evaluate their capabilities. However, concerns persist regarding the reliability of these benchmarks, which often lack clinical fidelity, robust data management, and safety-oriented evaluation metrics. To address these shortcomings, we introduce MedCheck, the first lifecycle-oriented assessment framework specifically designed for medical benchmarks. Our framework deconstructs a benchmark's development into five continuous stages, from design to governance, and provides a comprehensive checklist of 46 medically-tailored criteria. Using MedCheck, we conducted an in-depth empirical evaluation of 53 medical LLM benchmarks. Our analysis uncovers widespread, systemic issues, including a profound disconnect from clinical practice, a crisis of data integrity due to unmitigated contamination risks, and a systematic neglect of safety-critical evaluation dimensions like model robustness and uncertainty awareness. Based on these findings, MedCheck serves as both a diagnostic tool for existing benchmarks and an actionable guideline to foster a more standardized, reliable, and transparent approach to evaluating AI in healthcare.
years
2026 2verdicts
UNVERDICTED 2representative citing papers
Healthcare AI benchmarks show high scores on medical exams but sharply lower performance on real clinical tasks such as documentation and decision support, indicating a need for better frameworks to measure reliability and safety.
citing papers explorer
-
Self-Prompting Small Language Models for Privacy-Sensitive Clinical Information Extraction
Small open-weight language models can self-optimize prompts for clinical named entity recognition in dental notes, reaching micro F1 of 0.864 after DPO on Qwen2.5-14B.
-
Measuring What Matters: Benchmarking Generative, Multimodal, and Agentic AI in Healthcare
Healthcare AI benchmarks show high scores on medical exams but sharply lower performance on real clinical tasks such as documentation and decision support, indicating a need for better frameworks to measure reliability and safety.