REVIEW 8 cited by
A Comprehensive Survey on Evaluating Large Language Model Applications in the Medical Industry
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Since the inception of the Transformer architecture in 2017, Large Language Models (LLMs) such as GPT and BERT have evolved significantly, impacting various industries with their advanced capabilities in language understanding and generation. These models have shown potential to transform the medical field, highlighting the necessity for specialized evaluation frameworks to ensure their effective and ethical deployment. This comprehensive survey delineates the extensive application and requisite evaluation of LLMs within healthcare, emphasizing the critical need for empirical validation to fully exploit their capabilities in enhancing healthcare outcomes. Our survey is structured to provide an in-depth analysis of LLM applications across clinical settings, medical text data processing, research, education, and public health awareness. We begin by exploring the roles of LLMs in various medical applications, detailing their evaluation based on performance in tasks such as clinical diagnosis, medical text data processing, information retrieval, data analysis, and educational content generation. The subsequent sections offer a comprehensive discussion on the evaluation methods and metrics employed, including models, evaluators, and comparative experiments. We further examine the benchmarks and datasets utilized in these evaluations, providing a categorized description of benchmarks for tasks like question answering, summarization, information extraction, bioinformatics, information retrieval and general comprehensive benchmarks. This structure ensures a thorough understanding of how LLMs are assessed for their effectiveness, accuracy, usability, and ethical alignment in the medical domain. ...
Forward citations
Cited by 8 Pith papers
-
Every Sample Counts: Supervised Fine-Tuning of Language Models with Pointwise Constraints
Pointwise constrained fine-tuning via sample-wise augmented Lagrangians and learned relaxations reduces tail constraint violations across safety, tool-calling, and re-ranking while preserving average task performance.
-
RoboBrain 2.0 Technical Report
RoboBrain 2.0, a 7B/32B embodied vision-language model built on Qwen2.5-VL, reports state-of-the-art or near-top scores on several spatial and temporal reasoning benchmarks for robotics.
-
LIFEBench: Evaluating Length Instruction Following in Large Language Models
LIFEBench's evaluation of 26 LLMs shows most follow short length instructions but degrade sharply beyond a few hundred words, and none reliably hit vendor-claimed maximum output lengths.
-
MedGUIDE: Benchmarking Clinical Decision-Making in Large Language Models
MedGUIDE tests whether LLMs follow structured NCCN cancer-care decision trees and finds that even medical LLMs often lag general models on this task.
-
Q-FAKER: Query-free Hard Black-box Attack via Controlled Generation
Q-faker generates adversarial examples for NLU classifiers using surrogate-model gradients and controlled generation, requiring zero queries to the target.
-
Multiple Abstraction Level Retrieve Augment Generation
MAL-RAG retrieves document, section, paragraph, and multi-sentence chunks together and claims a 25.7% improvement in AI-judged answer correctness on glycoscience questions over single-level RAG.
-
The Road to Artificial SuperIntelligence: A Comprehensive Survey of Superalignment
A survey of scalable oversight for superalignment, reviewing weak-to-strong generalization, debate, RLAIF, and sandwiching, and concluding that current methods are not yet sufficient for superintelligent AI.
-
Survey of different Large Language Model Architectures: Trends, Benchmarks, and Challenges
A broad but error-prone survey of LLM and MLLM architectures, training methods, benchmarks, and challenges.
Discussion (0). Continue with ORCID to comment.