REVIEW 7 cited by
Self-Verification Improves Few-Shot Clinical Information Extraction
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Extracting patient information from unstructured text is a critical task in health decision-support and clinical research. Large language models (LLMs) have shown the potential to accelerate clinical curation via few-shot in-context learning, in contrast to supervised learning which requires much more costly human annotations. However, despite drastic advances in modern LLMs such as GPT-4, they still struggle with issues regarding accuracy and interpretability, especially in mission-critical domains such as health. Here, we explore a general mitigation framework using self-verification, which leverages the LLM to provide provenance for its own extraction and check its own outputs. This is made possible by the asymmetry between verification and generation, where the latter is often much easier than the former. Experimental results show that our method consistently improves accuracy for various LLMs in standard clinical information extraction tasks. Additionally, self-verification yields interpretations in the form of a short text span corresponding to each output, which makes it very efficient for human experts to audit the results, paving the way towards trustworthy extraction of clinical information in resource-constrained scenarios. To facilitate future research in this direction, we release our code and prompts.
Forward citations
Cited by 7 Pith papers
-
CollabLLM: From Passive Responders to Active Collaborators
CollabLLM computes multiturn-aware rewards by forward-simulating future user turns, then fine-tunes the LLM with RL to ask clarifying questions and guide users to their goals.
-
A Multi-Stage Large Language Model Framework for Extracting Suicide-Related Social Determinants of Health
A multi-stage LLM pipeline improves extraction of infrequent suicide-related social determinants from death narratives, but some evaluation results are compromised by using the test set to tune the system.
-
CardioCoT: Hierarchical Reasoning for Multimodal Survival Analysis
CardioCoT uses GPT-4o-generated hierarchical reasoning text as an extra input to a multimodal survival model, reporting higher C-index for MACE recurrence but with potential outcome-label leakage and no external validation.
-
Large Language Models for Scholarly Ontology Generation: An Extensive Analysis in the Engineering Field
Zero-shot LLMs, especially Claude 3 Sonnet and a fine-tuned 7B Mistral variant, classify semantic relations between engineering research topics with high F1 on the new IEEE-Rel-1K benchmark.
-
Scalable, Symbiotic, AI and Non-AI Agent Based Parallel Discrete Event Simulations
PDES orchestration of small language models with non-AI verifier agents raises accuracy on four toy tasks from about 23 percent to 68 percent, with the verifiers supplying most of the correctness.
-
Best Practices for Large Language Models in Radiology
The paper recommends starting LLM use in radiology with prompt optimization and retrieval augmentation, fine-tuning only when needed, and preferring locally hosted open models with human expert evaluation.
-
Generative AI in Medicine
A stakeholder-based review of generative AI use cases in medicine and the consent, privacy, transparency, hallucination, usability, equity, evaluation, and accountability challenges that stand between prototypes and s...
Discussion (0). Continue with ORCID to comment.