REVIEW 17 cited by
Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicine
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Generalist foundation models such as GPT-4 have displayed surprising capabilities in a wide variety of domains and tasks. Yet, there is a prevalent assumption that they cannot match specialist capabilities of fine-tuned models. For example, most explorations to date on medical competency benchmarks have leveraged domain-specific training, as exemplified by efforts on BioGPT and Med-PaLM. We build on a prior study of GPT-4's capabilities on medical challenge benchmarks in the absence of special training. Rather than using simple prompting to highlight the model's out-of-the-box capabilities, we perform a systematic exploration of prompt engineering. We find that prompting innovation can unlock deeper specialist capabilities and show that GPT-4 easily tops prior leading results for medical benchmarks. The prompting methods we explore are general purpose, and make no specific use of domain expertise, removing the need for expert-curated content. Our experimental design carefully controls for overfitting during the prompt engineering process. We introduce Medprompt, based on a composition of several prompting strategies. With Medprompt, GPT-4 achieves state-of-the-art results on all nine of the benchmark datasets in the MultiMedQA suite. The method outperforms leading specialist models such as Med-PaLM 2 by a significant margin with an order of magnitude fewer calls to the model. Steering GPT-4 with Medprompt achieves a 27% reduction in error rate on the MedQA dataset over the best methods to date achieved with specialist models and surpasses a score of 90% for the first time. Beyond medical problems, we show the power of Medprompt to generalize to other domains and provide evidence for the broad applicability of the approach via studies of the strategy on exams in electrical engineering, machine learning, philosophy, accounting, law, nursing, and clinical psychology.
Forward citations
Cited by 17 Pith papers
-
A Clinically Validated Foundation Model for Comprehensive Lung Pathology Interpretation
PulmoFoundation achieves 92.3% average AUC on 32 lung pathology tasks in prospective validation and raises pathologist accuracy from 83.8% to 91.7% in a crossover RCT.
-
PatientAgentBench: A Benchmark Framework for Evaluating Patient-Facing Health AI Agents
A clinician-validated benchmark for patient-facing health AI agents shows that even frontier models fail triage in up to a quarter of realistic tool-using conversations.
-
In-Context Learning for Wound Classification with Small Multimodal Language Models
Retrieval-based in-context learning, not zero-shot prompting, drives wound-classification gains in small multimodal models, with Qwen 3.5 27B reaching 0.872 accuracy on Kaggle and 0.678 on Medetec.
-
LLM Agents for Deliberative Collaboration: A Study on Joint Decision Making Under Partial Observability
A benchmark for LLM agents in partially observable joint decision-making reveals that deliberation challenges current models but can enable reflection and error correction.
-
Group Selection as a Safeguard Against AI Substitution
In an agent-based model of cumulative cultural evolution, AI-substitute strategies win under individual selection, but AI-complement strategies can spread when group boundaries are strong.
-
Toward a Vision-Language Foundation Model for Medical Data: Multimodal Dataset and Benchmarks for Vietnamese PET/CT Report Generation
A new Vietnamese PET/CT-report dataset improves medical VLM report generation and VQA, but clinical F1 scores remain modest.
-
Teaching large language models to reason like expert diagnosticians
An LLM agent and a 10-task benchmark built from 7,102 NEJM clinicopathologic cases push medical AI evaluation beyond final diagnosis accuracy.
-
Testing for LLM response differences: the case of a composite null consisting of semantically irrelevant query perturbations
A new hypothesis test for binary LLM responses treats semantically equivalent query perturbations as an unknown null set and gives asymptotic validity and consistency guarantees under a uniformity assumption.
-
HIVMedQA: Benchmarking large language models for HIV medical decision support
The HIVMedQA benchmark finds that Gemini 2.5 Pro leads on most clinical reasoning dimensions, medical fine-tuning does not guarantee gains, and LLM judges are more informative than lexical overlap.
-
Dissecting Clinical Reasoning in Language Models: A Comparative Study of Prompts and Model Adaptation Strategies
Prompt structure and LoRA adaptation both strongly affect F1 on NLI4CT clinical NLI, but the claimed consistent +8 to 12 point LoRA gains and over 97% validity are not supported by the reported per-configuration results.
-
Toward Trustworthy Large Language Model Agents in Healthcare
A scoped LLM agent with deterministic safety short-circuits and schema-validated tools automates healthcare appointment logistics at 91.8% success and $0.0324 per booking on synthetic tests.
-
Cross-Platform Evaluation of Large Language Model Safety in Pediatric Consultations: Evolution of Adversarial Robustness and the Scale Paradox
A three-model pediatric safety comparison finds smaller models score higher than Llama-3.3-70B, but the adversarial-robustness and evolution claims are undermined by an unexplained 30-query adversarial subset and non-...
-
The Non-Determinism of Small LLMs: Evidence of Low Answer Consistency in Repetition Trials of Standard Multiple-Choice Benchmarks
Small open-source LLMs give inconsistent answers on 20% to 50% of multiple-choice questions even at low temperature, while 50B-80B models are consistent on over 90% of questions.
-
Towards Effective Complementary Security Analysis using Large Language Models
Using Chain-of-Thought and Self-Consistency prompts, some LLMs removed over half of SAST false positives on a benchmark while missing no genuine weaknesses, and ensembling three models removed about 79%.
-
DeepLens Diagnosis Agent: Agentic Workflow Design Lets a Small Reasoning Model Compete with Frontier LLMs
A 7B medical model wrapped in a five-stage diagnosis workflow beats frontier LLMs on DiagnosisArena, gaining +36 points over its own single-shot baseline.
-
ALIGN: Prompt-based Attribute Alignment for Reliable, Responsible, and Personalized LLM-based Decision-Making
ALIGN is an open-source framework that aligns large language models to user-selected attributes via prompts and compares alignment methods across two decision-making domains.
-
Knowledge Augmented Finetuning Matters in both RAG and Agent Based Dialog Systems
On MobileCS2, a GPT-2 finetuned with retrieved knowledge outperforms a prompted GPT-3.5 in both RAG and agent dialog systems, though the comparison is confounded.
Discussion (0). Sign in to comment.