Pith. sign in

REVIEW 31 cited by

Large Language Models Encode Clinical Knowledge

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2212.13138 v1 pith:SPPDH3CO submitted 2022-12-26 cs.CL

Karan Singhal , Shekoofeh Azizi , Tao Tu , S. Sara Mahdavi , Jason Wei , Hyung Won Chung , Nathan Scales , Ajay Tanwani
show 22 more authors
Heather Cole-Lewis Stephen Pfohl Perry Payne Martin Seneviratne Paul Gamble Chris Kelly Nathaneal Scharli Aakanksha Chowdhery Philip Mansfield Blaise Aguera y Arcas Dale Webster Greg S. Corrado Yossi Matias Katherine Chou Juraj Gottweis Nenad Tomasev Yun Liu Alvin Rajkomar Joelle Barral Christopher Semturs Alan Karthikesalingam Vivek Natarajan
This is my paper · ORCID
classification cs.CL
keywords clinicalmedicalmodelsmodelevaluationflan-palmhumanknowledge
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Large language models (LLMs) have demonstrated impressive capabilities in natural language understanding and generation, but the quality bar for medical and clinical applications is high. Today, attempts to assess models' clinical knowledge typically rely on automated evaluations on limited benchmarks. There is no standard to evaluate model predictions and reasoning across a breadth of tasks. To address this, we present MultiMedQA, a benchmark combining six existing open question answering datasets spanning professional medical exams, research, and consumer queries; and HealthSearchQA, a new free-response dataset of medical questions searched online. We propose a framework for human evaluation of model answers along multiple axes including factuality, precision, possible harm, and bias. In addition, we evaluate PaLM (a 540-billion parameter LLM) and its instruction-tuned variant, Flan-PaLM, on MultiMedQA. Using a combination of prompting strategies, Flan-PaLM achieves state-of-the-art accuracy on every MultiMedQA multiple-choice dataset (MedQA, MedMCQA, PubMedQA, MMLU clinical topics), including 67.6% accuracy on MedQA (US Medical License Exam questions), surpassing prior state-of-the-art by over 17%. However, human evaluation reveals key gaps in Flan-PaLM responses. To resolve this we introduce instruction prompt tuning, a parameter-efficient approach for aligning LLMs to new domains using a few exemplars. The resulting model, Med-PaLM, performs encouragingly, but remains inferior to clinicians. We show that comprehension, recall of knowledge, and medical reasoning improve with model scale and instruction prompt tuning, suggesting the potential utility of LLMs in medicine. Our human evaluations reveal important limitations of today's models, reinforcing the importance of both evaluation frameworks and method development in creating safe, helpful LLM models for clinical applications.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 31 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. OpenAlex reports about 260 citations worldwide. Full citation record

  1. MedRealMM: A Real-World Multimodal Benchmark for Chinese Online Medical Consultation

    cs.AI 2026-07 conditional novelty 6.5 of 10

    On a new benchmark of 5,620 real multimodal online consultations, top LLMs trail the original physicians mainly because they trigger more unsafe or unsupported negative criteria.

  2. EpiBench: Can LLMs Understand Epitopes for Antibody Drug Discovery?

    cs.CL 2026-08 conditional novelty 6.0 of 10

    A new closed-book, sequence-only benchmark, EpiBench, measures epitope reasoning in LLMs and finds them near chance on residue-level localization and escape assessment, with only coarse region-level signal.

  3. CardioBench: A Real-World Data Benchmark for Evaluating Large Language Models in Clinically Authentic Cardiovascular Care Scenarios

    cs.CL 2026-07 conditional novelty 6.0 of 10

    MyoCardBench, a 13-task real-world cardiology benchmark, shows GPT-5.4 leading seven LLMs while ECG reading and clinical ethics remain near-random.

  4. PertReason: A Knowledge-Grounded Benchmark and Framework for Cell-State-Conditioned Mechanistic Reasoning of Perturbation Effects

    cs.LG 2026-07 conditional novelty 6.0 of 10

    PertReasonQA scores AI models on cell-state-conditioned mechanistic reasoning about perturbation effects, and PertReasonLM, trained with reasoning supervision, reaches 0.736 balanced accuracy and 0.976 edge recall ver...

  5. CausalFlip: A Benchmark for LLM Causal Judgment Beyond Semantic Matching

    cs.AI 2026-02 conditional novelty 6.0 of 10

    A new benchmark and training strategy show LLMs trained to internalize causal reasoning steps are less fooled by semantically similar, label-flipped questions than models using explicit chain-of-thought.

  6. Embedding Poisoning: Bypassing Safety Alignment via Embedding Semantic Shift

    cs.CR 2025-09 conditional novelty 6.0 of 10

    Small single-dimension perturbations to embeddings of high-risk tokens can flip aligned LLM responses from refusal to harmful output, and a search algorithm (SEP) locates such perturbations across models.

  7. Essential-Web v1.0: 24T tokens of organized web data

    cs.CL 2025-06 conditional novelty 6.0 of 10

    A 24T-token web corpus with automatic document-level taxonomy labels enables competitive domain-specific datasets via simple filters.

  8. Optimization-Inspired Few-Shot Adaptation for Large Language Models

    cs.LG 2025-05 conditional novelty 6.0 of 10

    OFA tunes LayerNorm parameters as optimization preconditioners and adds step-ratio and sharpness penalties, reporting consistent few-shot accuracy gains over baselines on Llama and GPT-2 models.

  9. Learnware of Language Models: Specialized Small Language Models Can Do Big

    cs.LG 2025-05 conditional novelty 6.0 of 10

    A system that matches specialized 8B-parameter language models to user tasks by comparing compact parameter-vector specifications beats 70B+ general LLMs on finance and medical benchmarks.

  10. Detecting and Mitigating Hateful Content in Multimodal Memes with Vision-Language Models

    cs.CV 2025-04 conditional novelty 6.0 of 10

    A prompt-based pipeline with GPT-4o detects hateful memes at state-of-the-art zero-shot accuracy and mitigates them by replacing hateful text or images, with 88% of 631 human-rated mitigated memes judged non-hateful.

  11. DMind Benchmark: Toward a Holistic Assessment of LLM Capabilities across the Web3 Domain

    cs.CR 2025-04 reject novelty 6.0 of 10

    DMind is a 3,543-item, nine-domain benchmark for LLMs in Web3; evaluation of 31 models shows strong fundamentals but weak security, token economics, and meme-related reasoning.

  12. Hypothesis Generation for Materials Discovery and Design Using Goal-Driven and Constraint-Guided LLM Agents

    cs.CL 2025-01 conditional novelty 6.0 of 10

    An LLM multi-agent framework with critic feedback and knowledge-graph grounding achieves higher scores on a new 50-paper benchmark for materials hypothesis generation, but the evaluation is LLM-based and human validat...

  13. TRIDENT: Benchmarking LLM Safety in Finance, Medicine, and Law

    cs.CL 2025-07 conditional novelty 5.0 of 10

    Trident-Bench provides 2,652 professionally validated harmful prompts across finance, law, and medicine, and shows that domain-specialized LLMs often comply with unethical requests more than generalist models.

  14. CLONE: Customizing LLMs for Efficient Latency-Aware Inference at the Edge

    cs.AR 2025-06 conditional novelty 5.0 of 10

    CLONE combines generative pruning, LoRA adapters, a parameter-free Mixture-of-Experts router, and learning-based DVFS to make LLM inference on edge devices faster and more energy-efficient, claiming up to 11.92x speed...

  15. Teaching with Lies: Curriculum DPO on Synthetic Negatives for Hallucination Detection

    cs.CL 2025-05 conditional novelty 5.0 of 10

    Using hallucinated benchmark answers as rejected DPO pairs, ordered by an external fact-checker's grounding score, improves hallucination detection in 1B-3B Llama models.

  16. Towards medical AI misalignment: a preliminary study

    cs.CY 2025-05 conditional novelty 5.0 of 10

    A custom role-playing prompt called the Goofy Game made four major LLMs produce plausible but incorrect medical recommendations.

  17. RADAR: Enhancing Radiology Report Generation with Supplementary Knowledge Injection

    cs.CV 2025-05 conditional novelty 5.0 of 10

    RADAR filters an LLM's radiology findings by agreement with an expert classifier and retrieves only the missing observations, reporting improved clinical accuracy on three datasets.

  18. Truth Neurons

    cs.CL 2025-05 reject novelty 5.0 of 10

    Language models contain small neuron subsets whose suppression degrades factuality performance, interpreted by the authors as 'truth neurons'.

  19. STAYKATE: Hybrid In-Context Example Selection Combining Representativeness Sampling and Retrieval-based Approach -- A Case Study on Science Domains

    cs.CL 2024-12 conditional novelty 5.0 of 10

    STAYKATE blends static representativeness sampling with kNN retrieval to pick in-context examples for LLM scientific NER, reporting modest F1 improvements over KATE and random baselines.

  20. Testing Medical Rules Web Services in Practice

    cs.SE 2024-12 conditional novelty 5.0 of 10

    EvoGURI, a domain-specific extension of EvoMaster, achieved top rule coverage and near-perfect mutation scores when testing a medical rule engine with a strict OpenAPI schema.

  21. SynEHRgy: Synthesizing Mixed-Type Structured Electronic Health Records using Decoder-Only Transformers

    cs.LG 2024-11 conditional novelty 5.0 of 10

    SynEHRgy tokenizes mixed-type MIMIC-III records into one sequence and trains a small decoder-only transformer to generate new synthetic patient records.

  22. Open-Source Agentic Hybrid RAG Framework for Scientific Literature Review

    cs.IR 2025-07 conditional novelty 4.0 of 10

    A DPO-tuned agentic hybrid RAG system that routes queries between a knowledge graph and a vector store beat a static baseline on a self-generated benchmark.

  23. Prompt Mechanisms in Medical Imaging: A Comprehensive Survey

    eess.IV 2025-06 conditional novelty 4.0 of 10

    A broad survey that organizes prompt mechanisms for medical image generation, segmentation, and classification into a two-dimensional taxonomy of core technologies and clinical applications.

  24. Balancing Knowledge Delivery and Emotional Comfort in Healthcare Conversational Systems

    cs.CL 2025-06 conditional novelty 4.0 of 10

    Fine-tuning a 1B medical chatbot on LLM-rewritten emotional dialogues improves its emotion scores with only small changes in n-gram overlap with the original medical responses.

  25. Open Foundation Models in Healthcare: Challenges, Paradoxes, and Opportunities with GenAI Driven Personalized Prescription

    cs.CL 2025-02 conditional novelty 4.0 of 10

    Open LLMs (LLaMA-2, LLaMA-3, Mistral, Meditron) roughly match GPT-4 on a 25-patient prescription-suitability check when given SmPC context via RAG, though some interaction classes degrade with RAG.

  26. Can ChatGPT Overcome Behavioral Biases in the Financial Sector? Classify-and-Rethink: Multi-Step Zero-Shot Reasoning in the Gold Investment

    q-fin.ST 2024-11 reject novelty 4.0 of 10

    A 'Classify-and-Rethink' prompt for ChatGPT produced higher backtested returns on gold trading than simpler prompts or buy-and-hold, though the comparison is confounded.

  27. BioPars: A Pretrained Biomedical Large Language Model for Persian Biomedical Text Mining

    cs.CL 2025-06 reject novelty 3.0 of 10

    A proposed Persian biomedical LLM, BioPars, is evaluated on medical QA datasets and reported to beat GPT-4 on a self-built Persian QA benchmark, but the training setup is not described.

  28. MedBioLM: Optimizing Medical and Biological QA with Fine-Tuned Large Language Models and Retrieval-Augmented Generation

    cs.CL 2025-02 reject novelty 3.0 of 10

    Fine-tuning GPT-4o on biomedical QA datasets improves accuracy on MedQA, PubMedQA, and BioASQ, while RAG adds little once fine-tuning is applied.

  29. A Survey on Large Language Models with some Insights on their Capabilities and Limitations

    cs.CL 2025-01 unverdicted novelty 3.0 of 10

    A broad survey of LLM methods and applications, plus an empirical section on how code-rich pretraining may influence chain-of-thought reasoning, the details of which are not visible in the supplied text.

  30. Cloud Platforms for Developing Generative AI Solutions: A Scoping Review of Tools and Services

    cs.DC 2024-12 conditional novelty 1.0 of 10

    A scoping review that aggregates and compares major cloud providers' generative AI tools and services, with no new empirical results.

  31. Survey of different Large Language Model Architectures: Trends, Benchmarks, and Challenges

    cs.LG 2024-12 conditional

    A broad but error-prone survey of LLM and MLLM architectures, training methods, benchmarks, and challenges.

Pith tools