REVIEW 6 cited by
OLAPH: Improving Factuality in Biomedical Long-form Question Answering
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
In the medical domain, numerous scenarios necessitate the long-form generation ability of large language models (LLMs). Specifically, when addressing patients' questions, it is essential that the model's response conveys factual claims, highlighting the need for an automated method to evaluate those claims. Thus, we introduce MedLFQA, a benchmark dataset reconstructed using long-form question-answering datasets related to the biomedical domain. We use MedLFQA to facilitate a cost-effective automatic evaluations of factuality. We also propose OLAPH, a simple and novel framework that utilizes cost-effective and multifaceted automatic evaluation to construct a synthetic preference set and answers questions in our preferred manner. Our framework leads us to train LLMs step-by-step to reduce hallucinations and include crucial medical claims. We highlight that, even on evaluation metrics not used during training, LLMs trained with our OLAPH framework demonstrate significant performance improvement in factuality. Our findings reveal that a 7B LLM trained with our OLAPH framework can provide long answers comparable to the medical experts' answers in terms of factuality. We believe that our work could shed light on gauging the long-text generation ability of LLMs in the medical domain. Our code and datasets are available.
Forward citations
Cited by 6 Pith papers
-
Automatic Evaluation of Healthcare LLMs Beyond Question-Answering
In healthcare LLM evaluation, multiple-choice accuracy and open-ended task scores correlate only weakly, and the paper's proposed Relaxed Perplexity metric aims to improve open-ended factuality scoring but rests on an...
-
Response Quality Assessment for Retrieval-Augmented Generation via Conditional Conformal Factuality
Conformal-RAG applies conformal prediction with a retrieval-based relevance score to guarantee the factuality of retained sub-claims in RAG responses.
-
Large Language Models for Cancer Communication: Evaluating Linguistic Quality, Safety, and Accessibility in Generative AI
General-purpose LLMs outperformed specialized medical LLMs on linguistic quality and emotional engagement for breast and cervical cancer questions, while medical models were simpler to read but scored worse on safety.
-
Harnessing RLHF for Robust Unanswerability Recognition and Trustworthy Response Generation in LLMs
SALU, a multi-task fine-tuning and confidence-guided RLHF method, reduces hallucinated answers on unanswerable Chinese CIR questions to 1.3 percent on the authors' private dataset.
-
The Aloe Family Recipe for Open and Specialized Healthcare LLMs
Aloe Beta, a family of open-weights health LLMs built from Llama 3.1 and Qwen 2.5, matches or exceeds closed medical models on MCQA benchmarks while improving safety via DPO.
-
MedBioLM: Optimizing Medical and Biological QA with Fine-Tuned Large Language Models and Retrieval-Augmented Generation
Fine-tuning GPT-4o on biomedical QA datasets improves accuracy on MedQA, PubMedQA, and BioASQ, while RAG adds little once fine-tuning is applied.
Discussion (0). Continue with ORCID to comment.