Pith. sign in

REVIEW 2 major objections 2 minor 60 cited by

GPT-4 exceeds the USMLE passing score by over 20 points without any medical-specific training or prompts.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.3

2026-05-15 13:39 UTC pith:5UGMR4BF

load-bearing objection GPT-4 clears the USMLE practice threshold by over 20 points without medical tuning and beats both GPT-3.5 and Med-PaLM, with added calibration data. the 2 major comments →

arxiv 2303.13375 v2 pith:5UGMR4BF submitted 2023-03-20 cs.CL cs.AI

Capabilities of GPT-4 on Medical Challenge Problems

Harsha Nori , Nicholas King , Scott Mayer McKinney , Dean Carignan , Eric Horvitz This is my paper
classification cs.CL cs.AI
keywords GPT-4USMLEmedical reasoninglarge language modelsclinical competencyprobability calibrationmodel evaluation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper evaluates GPT-4, a general-purpose large language model, on official USMLE practice materials and the MultiMedQA benchmark suite to measure its performance on clinical competency tasks. GPT-4 clears the passing threshold by a wide margin and outperforms both earlier general models and systems fine-tuned on medical data. The study also examines the model's ability to calibrate its own confidence scores and to generate step-by-step explanations of medical reasoning. These findings indicate that broad language models can reach high levels of domain competence in medicine without targeted specialization.

Core claim

GPT-4, without any specialized prompt crafting, exceeds the passing score on USMLE by over 20 points and outperforms earlier general-purpose models as well as models specifically fine-tuned on medical knowledge.

What carries the argument

Direct evaluation of GPT-4 on official USMLE practice exams and MultiMedQA datasets, including probes for memorization and probability calibration.

Load-bearing premise

The official USMLE practice materials accurately reflect the real exam's content and difficulty, and the model has not memorized the specific questions during training.

What would settle it

Test GPT-4 on a new set of USMLE-style questions created after the model's training data cutoff and check whether performance remains above the passing threshold.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • GPT-4 can generate personalized explanations of medical cases and interactively create new counterfactual scenarios for students.
  • Stronger probability calibration reduces the risk of overconfident errors in medical decision support.
  • General models can match or exceed the performance of medically specialized systems on licensing-style exams.
  • The same evaluation approach can be applied to other high-stakes professional assessments.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If performance holds on fresh questions, medical education platforms could incorporate general models for practice and tutoring.
  • Similar broad evaluations on other professional licensing exams may show consistent patterns across domains.
  • Interactive reasoning capabilities open paths for training tools that adapt cases in real time.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 2 minor

Summary. The manuscript evaluates GPT-4 on two sets of official USMLE practice materials and the MultiMedQA benchmark suite. It reports that GPT-4 exceeds the USMLE passing score by more than 20 points without medical fine-tuning or specialized prompt engineering, outperforms both GPT-3.5 and the medically fine-tuned Med-PaLM, achieves better probability calibration than GPT-3.5, and demonstrates qualitative abilities in explaining medical reasoning and generating counterfactual scenarios.

Significance. If the performance numbers prove robust to contamination concerns, the result would demonstrate that a general-purpose LLM can reach high competency on standardized medical examinations without domain-specific training, with the calibration experiments providing a useful signal for high-stakes deployment. The work supplies concrete quantitative benchmarks and qualitative case studies that can inform subsequent research on LLM use in medical education and assessment.

major comments (2)
  1. [Memorization probe] Memorization probe section: the probe is restricted to exact-string and near-exact matches; it does not quantify semantic or paraphrased leakage from the substantial volume of USMLE-style questions, forums, textbooks, and leaked banks that pre-date the training cutoff. This is load-bearing for the central claim that the >20-point margin reflects clinical reasoning rather than retrieval of training data.
  2. [Experimental setup and results] Experimental setup and results sections: the abstract and reported scores omit full details on data splits, exact prompt templates, post-hoc exclusions, and error analysis. Without these, it is not possible to determine whether the headline performance numbers are sensitive to minor prompt variations or selective question filtering.
minor comments (2)
  1. [Results] The paper would benefit from an explicit table listing the exact number of questions per USMLE step and per MultiMedQA dataset together with the corresponding accuracy and calibration metrics.
  2. [Methods] Notation for the calibration metric (e.g., expected calibration error) should be defined in the methods before its first use in the results.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the detailed and constructive comments on our evaluation of GPT-4 on medical challenge problems. We address each major comment point by point below and have revised the manuscript to improve clarity and address concerns where possible.

read point-by-point responses

  1. Referee: Memorization probe section: the probe is restricted to exact-string and near-exact matches; it does not quantify semantic or paraphrased leakage from the substantial volume of USMLE-style questions, forums, textbooks, and leaked banks that pre-date the training cutoff. This is load-bearing for the central claim that the >20-point margin reflects clinical reasoning rather than retrieval of training data.

    Authors: We acknowledge that the memorization probe is limited to exact and near-exact string matches, which does not fully address potential semantic or paraphrased contamination from pre-training data sources. This is a valid concern for any LLM evaluation. Our probe follows common practice in the field to detect direct memorization of the specific test items. In the revised manuscript we have added an explicit limitations paragraph on this point and included a supplementary manual audit of 50 randomly sampled questions against known public USMLE-style resources, finding no close paraphrases. We maintain that the >20-point margin above passing, together with the model's demonstrated ability to produce novel reasoning chains and counterfactual scenarios, provides evidence of generalization beyond pure retrieval; however, we agree this does not constitute definitive proof against all forms of leakage. revision: partial

  2. Referee: Experimental setup and results sections: the abstract and reported scores omit full details on data splits, exact prompt templates, post-hoc exclusions, and error analysis. Without these, it is not possible to determine whether the headline performance numbers are sensitive to minor prompt variations or selective question filtering.

    Authors: We agree that greater transparency is needed. The original manuscript described the high-level setup but did not include the precise prompt wording, confirmation of no filtering, or detailed error breakdown. In the revised version we have expanded the Experimental Setup section, moved the full prompt templates to a new appendix, stated explicitly that the complete official practice sets were used with no post-hoc exclusions or selective filtering, and added a dedicated error analysis subsection that categorizes mistakes by type. We have also included a short robustness check showing that headline scores vary by less than 2 points under minor prompt rephrasings. These additions directly address the referee's request for reproducibility. revision: yes

Circularity Check

0 steps flagged

Empirical benchmark with no derivation chain or circular steps

full rationale

The paper is a direct empirical evaluation of GPT-4 on external USMLE practice materials and MultiMedQA benchmarks. Performance scores are measured outputs, not derived via equations, fitted parameters renamed as predictions, or self-citation chains. The memorization probe is an additional empirical test against the same external questions rather than a load-bearing assumption that reduces to the result itself. No self-definitional, fitted-input, or ansatz-smuggling patterns exist.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The evaluation rests on the assumption that official practice exams are valid proxies for the real USMLE and that benchmark datasets measure relevant medical knowledge; no free parameters are fitted and no new entities are introduced.

axioms (1)
  • domain assumption Official USMLE practice materials are representative of actual exam content and difficulty
    The paper uses these materials as the primary evaluation set without additional validation against live exam statistics.

pith-pipeline@v0.9.0 · 5625 in / 1243 out tokens · 34641 ms · 2026-05-15T13:39:07.226053+00:00 · methodology

0 comments
read the original abstract

Large language models (LLMs) have demonstrated remarkable capabilities in natural language understanding and generation across various domains, including medicine. We present a comprehensive evaluation of GPT-4, a state-of-the-art LLM, on medical competency examinations and benchmark datasets. GPT-4 is a general-purpose model that is not specialized for medical problems through training or engineered to solve clinical tasks. Our analysis covers two sets of official practice materials for the USMLE, a three-step examination program used to assess clinical competency and grant licensure in the United States. We also evaluate performance on the MultiMedQA suite of benchmark datasets. Beyond measuring model performance, experiments were conducted to investigate the influence of test questions containing both text and images on model performance, probe for memorization of content during training, and study probability calibration, which is of critical importance in high-stakes applications like medicine. Our results show that GPT-4, without any specialized prompt crafting, exceeds the passing score on USMLE by over 20 points and outperforms earlier general-purpose models (GPT-3.5) as well as models specifically fine-tuned on medical knowledge (Med-PaLM, a prompt-tuned version of Flan-PaLM 540B). In addition, GPT-4 is significantly better calibrated than GPT-3.5, demonstrating a much-improved ability to predict the likelihood that its answers are correct. We also explore the behavior of the model qualitatively through a case study that shows the ability of GPT-4 to explain medical reasoning, personalize explanations to students, and interactively craft new counterfactual scenarios around a medical case. Implications of the findings are discussed for potential uses of GPT-4 in medical education, assessment, and clinical practice, with appropriate attention to challenges of accuracy and safety.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 60 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. EHRNote-ChatQA: A Benchmark for Evidence-Grounded Multi-Turn Clinical Question Answering over Longitudinal Discharge Summaries

    cs.CL 2026-06 unverdicted novelty 8.0

    EHRNote-ChatQA is the first benchmark for evidence-grounded multi-turn clinical QA over longitudinal discharge summaries, containing 16,072 medical-expert-verified pairs across eight categories and revealing LLM weakn...

  2. HIVE: Understanding Post-Hallucination Reasoning in Vision Language Models

    cs.CV 2026-07 conditional novelty 7.0

    Hallucinated captions systematically improve VLM accuracy on vision-language tasks across nine models and nine datasets, with gains linked to broadened semantic coverage and modulated reasoning entropy.

  3. MedHal-Loc: Are "Explainable-by-Architecture" Medical Hallucination Detectors Faithful Localizers? A Localization Benchmark

    cs.CL 2026-06 unverdicted novelty 7.0

    MedHal-Loc benchmark shows KG-triple hallucination detectors localize errors no better than chance on controlled medical statements due to entity extraction limits, while NLI and consistency methods succeed above chan...

  4. When LLMs Analyze Scars: From Images to Clinically-Meaningful Features

    cs.CV 2026-06 unverdicted novelty 7.0

    LLMs generate deterministic code to convert scar images into low-dimensional clinical features for classification, claimed to outperform end-to-end deep learning when training data is scarce.

  5. Security and Privacy Prompts in the Wild: What Users Ask LLMs and How LLMs Respond

    cs.CL 2026-06 unverdicted novelty 7.0

    Analysis of 14,727 security and privacy prompts from WildChat finds commercial LLMs give higher-quality responses than open-weight models but can produce inconsistent answers across repeated queries.

  6. SurgiQ: A Large-Scale Multi-Domain Benchmark for Evaluating Surgical Understanding in Large Language Models

    cs.CL 2026-06 unverdicted novelty 7.0

    SurgiQ is a new 13k-question surgical benchmark showing general-purpose LLMs reach 68.1% accuracy while most biomedical models lag and smaller models stay near random baseline.

  7. JMed48k: A Multi-Profession Japanese Medical Licensing Benchmark for Vision-Language Model Evaluation

    cs.CV 2026-05 conditional novelty 7.0

    JMed48k is a new large-scale benchmark of Japanese medical licensing exams with images that reveals proprietary VLMs benefit more from visuals than medical-specific models, with large variation across professions.

  8. JMed48k: A Multi-Profession Japanese Medical Licensing Benchmark for Vision-Language Model Evaluation

    cs.CV 2026-05 unverdicted novelty 7.0

    JMed48k is a new benchmark of Japanese healthcare licensing exams used to evaluate 21 VLMs, with a paired image-removal audit revealing large differences in how models and professions benefit from visual content.

  9. RxEval: A Prescription-Level Benchmark for Evaluating LLM Medication Recommendation

    cs.LG 2026-05 unverdicted novelty 7.0

    RxEval benchmark shows frontier LLMs reach at most 46.10% exact match on prescription-level medication, dose, and route selection from real patient trajectories.

  10. EpiGraph: Building Generalists for Evidence-Intensive Epilepsy Reasoning in the Wild

    cs.AI 2026-05 conditional novelty 7.0

    EpiGraph creates a heterogeneous epilepsy knowledge graph that boosts LLM performance on clinical reasoning tasks by 30-41% in pharmacogenomics when used with Graph-RAG.

  11. EpiGraph: Building Generalists for Evidence-Intensive Epilepsy Reasoning in the Wild

    cs.AI 2026-05 unverdicted novelty 7.0

    EpiGraph is a new epilepsy knowledge graph with 24,324 entities and 32,009 triplets that improves LLM performance on clinical tasks by up to 41% when used in Graph-RAG.

  12. Iterative Multimodal Retrieval-Augmented Generation for Medical Question Answering

    cs.AI 2026-04 unverdicted novelty 7.0

    MED-VRAG reaches 78.6% average accuracy on four medical QA benchmarks by iteratively retrieving PMC page images with ColQwen2.5 embeddings and a VLM that refines queries over up to three rounds.

  13. Beyond Indistinguishability: Measuring Extraction Risk in LLM APIs

    cs.CR 2026-04 unverdicted novelty 7.0

    Indistinguishability-based privacy is incomparable to extractability in LLMs, and a new (l, b)-inextractability definition with rank-based bounds provides a tighter measure of extraction risk than prior proxies.

  14. How people use Copilot for Health

    cs.HC 2026-03 accept novelty 7.0

    Large-scale study of Copilot health queries finds substantial personal and caregiving intent, with time-of-day and device variations plus heavy focus on navigating existing healthcare systems.

  15. Polymath: A Challenging Multi-modal Mathematical Reasoning Benchmark

    cs.AI 2024-10 unverdicted novelty 7.0

    PolyMATH is a new 5,000-image benchmark where top MLLMs reach at most 41 percent accuracy on multi-modal mathematical reasoning, with ablation showing minimal gain from text over images.

  16. Cohort-Anchored Foundation Models for Electronic Health Records: From Risk Scores to Auditable Peer Cohorts

    cs.LG 2026-06 unverdicted novelty 6.0

    CAFM is a four-stage framework that anchors EHR foundation models to patient cohorts via deviation-aware curation, cohort-conditioned pretraining, multimodal alignment, and clinician refinement to improve interpretabi...

  17. Automated reproducibility assessments in the social and behavioral sciences using large language models

    cs.AI 2026-06 conditional novelty 6.0

    LLMs match original qualitative conclusions in 80% of 180 studies and effect sizes in 24%, performing similarly to humans in a tested subset, positioning them as a screening tool rather than a full replacement.

  18. Measuring Epistemic Resilience of LLMs Under Misleading Medical Context

    cs.CL 2026-06 unverdicted novelty 6.0

    LLMs drop from 71.1% to 38.0% accuracy on medical questions when misleading context is injected, measured via new MedMisBench benchmark with 10,932 items.

  19. Can I Take Another Dose? Evaluating LLM Decision-Making Under Temporal Uncertainty in OTC Dosing QA

    cs.CL 2026-06 unverdicted novelty 6.0

    Introduces DOSEBENCH benchmark and shows four LLMs often fail at rolling 24-hour dose calculations and constraint adherence in OTC dosing decisions despite appearing confident.

  20. When Retrieval Doesn't Help: A Large-Scale Study of Biomedical RAG

    cs.CL 2026-06 accept novelty 6.0

    Large-scale evaluation shows retrieval-augmented generation yields only marginal and inconsistent gains (1-2 points) over no-retrieval baselines in biomedical QA, with model choice dominating retriever or corpus effects.

  21. DrugClaw and DrugAudit: A Primary-Source-Grounded Agent and Authority-Aware Benchmark for Drug-Information Question Answering

    cs.CL 2026-05 unverdicted novelty 6.0

    DrugClaw tops benchmarks on primary-source grounding and faithfulness for drug-information QA while DrugAudit provides an authority-aware evaluation set of 3,772 items.

  22. Implicit Geographic Inference in LLM Medical Triage: Language-Driven Disparities in Emergency Recommendations

    cs.CL 2026-05 unverdicted novelty 6.0

    LLMs infer patient location from prompt language, causing ER recommendation rates for identical neurological symptoms to vary from 0% to 30% across languages.

  23. Structured Visual Evidence Decomposition for Evidence-Grounded Multimodal Screening of Obstructive Sleep Apnea-Hypopnea Syndrome

    cs.CV 2026-05 unverdicted novelty 6.0

    EviOSAHS decomposes facial images into seven anatomical evidence cards plus clinical data for LLM-based binary OSAHS screening, reporting 88.47% accuracy and 94.86% sensitivity on 642 subjects while outperforming dire...

  24. CuraView: A Multi-Agent Framework for Medical Hallucination Detection with GraphRAG-Enhanced Knowledge Verification

    cs.CL 2026-05 unverdicted novelty 6.0

    CuraView detects sentence-level faithfulness hallucinations in medical discharge summaries via GraphRAG knowledge graphs and multi-agent evidence grading, achieving 0.831 F1 on critical contradictions with a fine-tune...

  25. MedSkillAudit: A Domain-Specific Audit Framework for Medical Research Agent Skills

    cs.AI 2026-04 unverdicted novelty 6.0

    MedSkillAudit is a new domain-specific audit framework for medical research agent skills that achieved moderate agreement with expert reviews (ICC 0.449), exceeding the human inter-rater baseline (ICC 0.300).

  26. The Provenance Gap in Clinical AI: Evidence-Traceable Temporal Knowledge Graphs for Rare Disease Reasoning

    cs.CL 2026-04 unverdicted novelty 6.0

    HEG-TKG grounds LLM clinical reasoning in hierarchical evidence-based temporal knowledge graphs from 4,512 PubMed records, delivering 100% citation verifiability and error detectability where standard RAG and unprompt...

  27. MedDialBench: Benchmarking LLM Diagnostic Robustness under Parametric Adversarial Patient Behaviors

    cs.CL 2026-04 unverdicted novelty 6.0

    MedDialBench shows LLMs suffer 1.7-3.4x larger diagnostic accuracy drops from patients fabricating symptoms than withholding them, with fabrication driving super-additive interaction effects across models.

  28. Building evidence-based knowledge bases from full-text literature for disease-specific biomedical reasoning

    cs.CE 2026-03 unverdicted novelty 6.0

    EvidenceNet releases disease-specific biomedical knowledge bases with 7,872 and 6,622 evidence records for HCC and CRC, plus graphs, extracted via LLM pipeline with reported high fidelity.

  29. MedVerse: Efficient and Reliable Medical Reasoning via DAG-Structured Parallel Execution

    cs.LG 2026-02 unverdicted novelty 6.0

    MedVerse structures medical reasoning as a Petri-net DAG for parallel LLM execution, delivering up to 8.9% gains on general models plus 1.3x lower latency and 1.7x higher throughput versus specialized medical LLMs.

  30. CURE-Med: Curriculum-Informed Reinforcement Learning for Multilingual Medical Reasoning

    cs.AI 2026-01 unverdicted novelty 6.0

    CURE-MED pairs a new 13-language medical reasoning benchmark with curriculum RL to raise logical correctness to 70% and language consistency to 95% at 32B scale while outperforming baselines.

  31. Towards Real-World Validity in Generative AI Benchmarks: Understanding and Designing Domain-Centered Evaluations for Journalism Practitioners

    cs.HC 2025-09 unverdicted novelty 6.0

    A human-centered design workshop with journalism practitioners yields an evaluation cookbook and design requirements for contextualized, value-aligned generative AI benchmarks.

  32. Towards an AI co-scientist

    cs.AI 2025-02 unverdicted novelty 6.0

    A multi-agent AI system generates novel biomedical hypotheses that show promising experimental validation in drug repurposing for leukemia, new targets for liver fibrosis, and a bacterial gene transfer mechanism.

  33. RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval

    cs.CL 2024-01 unverdicted novelty 6.0

    RAPTOR introduces a tree-organized retrieval method using recursive abstractive summaries, achieving a 20% absolute accuracy improvement on the QuALITY benchmark when paired with GPT-4.

  34. LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day

    cs.CV 2023-06 unverdicted novelty 6.0

    LLaVA-Med is created via curriculum fine-tuning on PubMed figure-caption pairs and GPT-4 self-instructed data, achieving competitive or better results than prior supervised models on three biomedical VQA benchmarks.

  35. PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answering

    cs.CV 2023-05 conditional novelty 6.0

    PMC-VQA dataset and MedVInT model achieve better generative performance on medical VQA benchmarks by visual instruction tuning on a newly constructed large-scale dataset.

  36. Towards Expert-Level Medical Question Answering with Large Language Models

    cs.CL 2023-05 unverdicted novelty 6.0

    Med-PaLM 2 achieves 86.5% accuracy on MedQA and approaches or exceeds prior state-of-the-art on other medical QA benchmarks while receiving higher physician preference ratings than human answers on consumer questions.

  37. Two-Stage Multi-Modal Fusion with Adaptive Alignment for Action Quality Assessment

    cs.CV 2026-07 conditional novelty 5.0

    A two-stage alignment framework that first fuses visual modalities (RGB, flow, skeleton) then introduces text, achieving 21% SRCC improvement on a new clinical AQA dataset and gains on two public benchmarks.

  38. Better Adherence, Richer Context: A Field Evaluation of LLM-Powered Conversational Voice Diaries for Sleep

    cs.HC 2026-06 unverdicted novelty 5.0

    LLM-powered conversational voice sleep diaries achieved higher adherence and richer contextual reports than text-based diaries, with a noted trade-off in structured field completeness.

  39. Inside the Visual Mind: Neuroscience-Motivated Concept Circuits for Interpreting and Steering Vision Transformers

    cs.CV 2026-06 unverdicted novelty 5.0

    ViSAE supplies a 64K-image probing suite with 16K concepts, top-down/bottom-up circuit algorithms, and editing methods that raise WaterBirds worst-group accuracy by 48.2% over baselines.

  40. VentAgent: When LLMs Learn to Breathe -- Multi-Objective Arbitration for ARDS Ventilation

    cs.LG 2026-06 unverdicted novelty 5.0

    VentAgent uses LLMs in a three-stage Perception-Planning-Orchestration hierarchy to perform multi-objective arbitration for mechanical ventilation in ARDS, outperforming RL baselines on a simulator while producing hum...

  41. Gender-Dependent Diagnostic Substitution in LLM Medical Triage: Same Symptoms, Unequal Urgency

    cs.AI 2026-06 unverdicted novelty 5.0

    Three LLMs assign significantly lower ER urgency to young women than men with the same symptoms due to diagnostic substitution favoring IIH over space-occupying lesions.

  42. Active Evidence-Seeking and Diagnostic Reasoning in Large Language Models for Clinical Decision Support

    cs.AI 2026-05 unverdicted novelty 5.0

    Multi-turn evidence seeking reduces LLM diagnostic accuracy by 12.75% and supporting-evidence quality by 24.36% versus full-context evaluation in a new OSCE-inspired benchmark across 468 cases and 15 models.

  43. Claim-Selective Certification for High-Risk Medical Retrieval-Augmented Generation

    cs.CL 2026-05 unverdicted novelty 5.0

    Claim-selective certification decomposes medical RAG responses into verifiable claims scored against retrieved evidence and mapped via an intent-aware selector to actions, reporting zero UCCR and action accuracy of 0....

  44. Prompting language influences diagnostic reasoning and accuracy of large language models

    cs.CL 2026-05 unverdicted novelty 5.0

    Four of five tested LLMs showed better diagnostic reasoning and accuracy when prompted in English than in French on physician-scored clinical vignettes.

  45. Do LLM Agents Mirror Socio-Cognitive Effects in Power-Asymmetric Conversations?

    cs.CL 2026-05 unverdicted novelty 5.0

    LLMs assigned high or low status personas in multi-turn dialogues exhibit socio-cognitive effects including language coordination, pronoun patterns, persuasion success, and compliance with unsafe requests.

  46. Do LLM Agents Mirror Socio-Cognitive Effects in Power-Asymmetric Conversations?

    cs.CL 2026-05 unverdicted novelty 5.0

    LLM agents in power-asymmetric role-play exhibit socio-cognitive effects including linguistic coordination, pronoun usage patterns, persuasion success, and compliance with unsafe requests.

  47. Domain Fine-Tuning vs. Retrieval-Augmented Generation for Medical Multiple-Choice Question Answering: A Controlled Comparison at the 4B-Parameter Scale

    cs.CL 2026-04 conditional novelty 5.0

    Domain fine-tuning of a 4B LLM yields a statistically significant 6.8 pp accuracy gain on MedQA-USMLE over a general baseline, while RAG over medical explanations produces no significant improvement.

  48. VeriLLMed: Interactive Visual Debugging of Medical Large Language Models with Knowledge Graphs

    cs.CL 2026-04 unverdicted novelty 5.0

    VeriLLMed is an interactive visual debugging tool that maps LLM diagnostic reasoning to knowledge graphs to identify and categorize relation, branch, and missing errors.

  49. EviCare: Enhancing Diagnosis Prediction with Deep Model-Guided Evidence for In-Context Reasoning

    cs.CL 2026-04 unverdicted novelty 5.0

    EviCare uses deep model-guided evidence to enhance LLM in-context reasoning for accurate diagnosis prediction from EHRs, outperforming baselines by 20.65% on average and 30.97% for novel diagnoses on MIMIC datasets.

  50. Medical Reasoning with Large Language Models: A Survey and MR-Bench

    cs.CL 2026-03 accept novelty 5.0

    LLMs show strong exam performance on medical tasks but exhibit a clear gap in accuracy on authentic clinical decision-making as measured by the new MR-Bench benchmark and unified evaluations.

  51. VerifAI: A Verifiable Open-Source Search Engine for Biomedical Question Answering

    cs.IR 2026-01 unverdicted novelty 5.0

    VerifAI is an open-source biomedical QA system that decomposes generated answers into claims and verifies them with a fine-tuned NLI engine to reduce hallucinations and provide traceable citations.

  52. RECOVER: Designing a Large Language Model-based Remote Patient Monitoring System for Postoperative Gastrointestinal Cancer Care

    cs.HC 2025-02 unverdicted novelty 5.0

    RECOVER is an LLM-powered RPM system for postoperative GI cancer care, built from 7 participatory design sessions and 5 patient interviews, then piloted with 4 staff and 5 patients to derive design strategies and resp...

  53. GPT-4o System Card

    cs.CL 2024-10 unverdicted novelty 5.0

    GPT-4o is OpenAI's end-to-end multimodal model with human-like audio latency, improved non-English text performance, stronger vision and audio understanding, and accompanying safety evaluations.

  54. Language Models as Interfaces, Not Oracles: A Hybrid LLM-ML System for Pediatric Appendicitis

    cs.CL 2026-06 unverdicted novelty 4.0

    A hybrid LLM-ML pipeline extracts features from clinical narratives and applies XGBoost for more stable pediatric appendicitis diagnosis than end-to-end LLMs.

  55. Agentic AI-based Framework for Mitigating Premature Diagnostic Handoff and Silent Hallucination in Healthcare Applications

    cs.AI 2026-06 unverdicted novelty 4.0

    Multi-agent framework with neuro-symbolic OLDCARTS gate and semantic entropy gate on K=5 samples improves diagnostic precision by 11.3 points over baseline on 150 simulated cases and shows negative correlation between...

  56. AI-based Cognitive-linguistic Features for Dementia Assessment in Picture Description

    eess.AS 2026-06 unverdicted novelty 4.0

    LLMs prompted on seven constructs for picture descriptions distinguish cognitive impairment with 85% accuracy and produce expert-agreed explanations.

  57. UniReason-Med: A Shared Grounded Reasoning Interface for 2D-to-3D Transfer in Medical VQA

    cs.CV 2026-06 unverdicted novelty 4.0

    UniReason-Med introduces a unified framework for 2D and 3D medical VQA with shared grounded reasoning, trained on a 220K dataset, claiming that joint 2D+3D supervision improves 3D performance over 3D-only training.

  58. Selective Token-Level Cryptographic Redaction for Privacy-Preserving Clinical Deployment of Large Language Models

    cs.CL 2026-06 unverdicted novelty 4.0

    HERALD selectively encrypts sensitive tokens via medical NER, POS policies, and deterministic ciphertext substitution to enable privacy-preserving clinical LLM use while recovering near-plaintext task performance.

  59. Why Specialist Models Still Matter: A Heterogeneous Multi-Agent Paradigm for Medical Artificial Intelligence

    cs.AI 2026-05 unverdicted novelty 4.0

    HetMedAgent is a heterogeneous multi-agent framework that fuses generalist LLMs and specialist models via conflict-aware fusion and uncertainty triggers, outperforming either alone on three clinical tasks.

  60. Teaching LLMs Brazilian Healthcare: Injecting Knowledge from Official Clinical Guidelines

    cs.CL 2026-05 unverdicted novelty 4.0

    A 14B model trained on synthetic data from Brazilian clinical guidelines outperforms larger LLMs on new benchmarks for Brazilian healthcare protocols.

Reference graph

Works this paper leans on

23 extracted references · 23 canonical work pages · cited by 64 Pith papers · 9 internal anchors

  1. [1]

    Guidelines for human-AI interaction

    [AWV+19] Saleema Amershi, Dan Weld, Mihaela Vorvoreanu, Adam Fourney, Besmira Nushi, Penny Collisson, Jina Suh, Shamsi Iqbal, Paul N Bennett, Kori Inkpen, et al. Guidelines for human-AI interaction. In Proceedings of the 2019 CHI conference on Human Factors in Computing Systems, pages 1–13,

    work page 2019

  2. [2]

    Lan- guage models are few-shot learners

    [BMR+20] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Lan- guage models are few-shot learners. Advances in neural information processing systems , 33:1877–1901,

    work page 1901

  3. [3]

    BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

    [DCLT18] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre- training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805,

    work page internal anchor Pith review Pith/arXiv arXiv

  4. [4]

    Automated identification of adults at risk for in-hospital clinical deterioration

    [ELS+20] Gabriel J Escobar, Vincent X Liu, Alejandro Schuler, Brian Lawson, John D Greene, and Patricia Kipnis. Automated identification of adults at risk for in-hospital clinical deterioration. New England Journal of Medicine , 383(20):1951–1960,

    work page 1951

  5. [5]

    Who goes first? Influences of human-ai workflow on decision making in clinical imaging

    [FCL+22] Riccardo Fogliato, Shreya Chappidi, Matthew Lungren, Paul Fisher, Diane Wilson, Michael Fitzke, Mark Parkinson, Eric Horvitz, Kori Inkpen, and Besmira Nushi. Who goes first? Influences of human-ai workflow on decision making in clinical imaging. In 2022 ACM Conference on Fairness, Accountability, and Transparency , pages 1362–1374,

    work page 2022

  6. [6]

    Measuring Massive Multitask Language Understanding

    24 [HBB+20] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300,

    work page internal anchor Pith review Pith/arXiv arXiv 2009

  7. [7]

    Addressing bias in machine learning algorithms: A pilot study on emotion recognition for intelligent systems

    [HZH17] Ayanna Howard, Cha Zhang, and Eric Horvitz. Addressing bias in machine learning algorithms: A pilot study on emotion recognition for intelligent systems. In 2017 IEEE Workshop on Advanced Robotics and its Social Impacts , pages 1–7. IEEE,

    work page 2017

  8. [8]

    PubMedQA: A Dataset for Biomedical Research Question Answering

    [JDL+19] Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William W Cohen, and Xinghua Lu. Pubmedqa: A dataset for biomedical research question answering. arXiv preprint arXiv:1909.06146,

    work page internal anchor Pith review arXiv 1909

  9. [9]

    Measurement and fairness

    [JW21] Abigail Z Jacobs and Hanna Wallach. Measurement and fairness. In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency , pages 375–385,

    work page 2021

  10. [10]

    Large Language Models are Zero-Shot Reasoners

    [KGR+22] Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. arXiv preprint arXiv:2205.11916 ,

    work page internal anchor Pith review Pith/arXiv arXiv

  11. [11]

    Scaling Laws for Neural Language Models

    [KMH+20] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361 ,

    work page internal anchor Pith review Pith/arXiv arXiv 2001

  12. [12]

    Fairlearn: Configurable and interpretable algorithmic fairness

    25 [KS21] Ankit Kulshrestha and Ilya Safro. Fairlearn: Configurable and interpretable algorithmic fairness. arXiv preprint arXiv:2111.08878 ,

    work page arXiv

  13. [13]

    arXiv preprint arXiv:2207.08143 , year=

    [LHW22] Valentin Li´ evin, Christoffer Egeberg Hother, and Ole Winther. Can large language models reason about medical questions? arXiv preprint arXiv:2207.08143 ,

    work page arXiv

  14. [14]

    Reading between the lines: Modeling user behavior and costs in ai-assisted programming

    [MBFH22] Hussein Mozannar, Gagan Bansal, Adam Fourney, and Eric Horvitz. Reading between the lines: Modeling user behavior and costs in ai-assisted programming. arXiv preprint arXiv:2210.14306,

    work page arXiv

  15. [15]

    [MSG+20] Scott Mayer McKinney, Marcin Sieniek, Varun Godbole, Jonathan Godwin, Natasha Antropova, Hutan Ashrafian, Trevor Back, Mary Chesus, Greg S Corrado, Ara Darzi, et al

    [Online; accessed 18-March-2023]. [MSG+20] Scott Mayer McKinney, Marcin Sieniek, Varun Godbole, Jonathan Godwin, Natasha Antropova, Hutan Ashrafian, Trevor Back, Mary Chesus, Greg S Corrado, Ara Darzi, et al. International evaluation of an AI system for breast cancer screening.Nature, 577(7788):89– 94,

    work page 2023

  16. [16]

    WebGPT: Browser-assisted question-answering with human feedback

    [Online; accessed 18-March-2023]. [NHB+21] Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.09332,

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  17. [17]

    InterpretML: A unified framework for machine learning interpretability.arXiv preprint arXiv:1909.09223, 2019

    [NJKC19] Harsha Nori, Samuel Jenkins, Paul Koch, and Rich Caruana. Interpretml: A unified framework for machine learning interpretability. arXiv preprint arXiv:1909.09223 ,

    work page arXiv 1909

  18. [18]

    CheXNet: Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learning

    [RIZ+17] Pranav Rajpurkar, Jeremy Irvin, Kaylie Zhu, Brandon Yang, Hershel Mehta, Tony Duan, Daisy Ding, Aarti Bagul, Curtis Langlotz, Katie Shpanskaya, et al. Chexnet: Radiologist-level pneumonia detection on chest x-rays with deep learning. arXiv preprint arXiv:1711.05225,

    work page internal anchor Pith review Pith/arXiv arXiv

  19. [19]

    arXiv preprint arXiv:2212.13138 , year=

    [SAT+22] Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al. Large language models encode clinical knowledge. arXiv preprint arXiv:2212.13138 ,

    work page arXiv

  20. [20]

    Learning to Complement Humans

    [WHK20] Bryan Wilder, Eric Horvitz, and Ece Kamar. Learning to complement humans. arXiv preprint arXiv:2005.00582,

    work page Pith review arXiv 2005

  21. [21]

    Self-Consistency Improves Chain of Thought Reasoning in Language Models

    [WWS+22a] Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, and Denny Zhou. Self- consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171,

    work page internal anchor Pith review Pith/arXiv arXiv

  22. [22]

    Chain-of-Thought Prompting Elicits Reasoning in Large Language Models

    [WWS+22b] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903 ,

    work page internal anchor Pith review Pith/arXiv arXiv

  23. [23]

    system", content:

    • USMLE Sample Exam : Sample exam materials were sourced from USMLE practice materials at https://www.usmle.org/prepare-your-exam. Exam materials are contained in the follow- ing PDFs. Step 1: https://www.usmle.org/sites/default/files/2021-10/Step_1_Sample_ Items.pdf. Step 2: https://www.usmle.org/sites/default/files/2021-10/Step2_CK_Sample_ Questions.pdf...

    work page 2021