Across two TREC shared-task years, top LLM systems matched human writers on factual accuracy and completeness but not on simplicity or brevity, while common automatic metrics correlated poorly with manual judgments.
UM_FHS at TREC 2024 PLABA: Exploration of Fine-tuning and AI agent approach for plain language adaptations of biomedical text
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
abstract
This paper describes our submissions to the TREC 2024 PLABA track with the aim to simplify biomedical abstracts for a K8-level audience (13-14 years old students). We tested three approaches using OpenAI's gpt-4o and gpt-4o-mini models: baseline prompt engineering, a two-AI agent approach, and fine-tuning. Adaptations were evaluated using qualitative metrics (5-point Likert scales for simplicity, accuracy, completeness, and brevity) and quantitative readability scores (Flesch-Kincaid grade level, SMOG Index). Results indicated that the two-agent approach and baseline prompt engineering with gpt-4o-mini models show superior qualitative performance, while fine-tuned models excelled in accuracy and completeness but were less simple. The evaluation results demonstrated that prompt engineering with gpt-4o-mini outperforms iterative improvement strategies via two-agent approach as well as fine-tuning with gpt-4o. We intend to expand our investigation of the results and explore advanced evaluations.
fields
cs.CL 1years
2025 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
Lessons from the TREC Plain Language Adaptation of Biomedical Abstracts (PLABA) track
Across two TREC shared-task years, top LLM systems matched human writers on factual accuracy and completeness but not on simplicity or brevity, while common automatic metrics correlated poorly with manual judgments.