REVIEW 5 major objections 6 minor 3 cited by
MedBioLM: Optimizing Medical and Biological QA with Fine-Tuned Large Language Models and Retrieval-Augmented Generation
T0 review · 5 major / 6 minor · reviewed 2026-08-09 · deepseek-v4-flash
Pith's one-line read Fine-tuning GPT-4o on biomedical QA data yields MedBioLM, which outperforms the base model on MedQA, PubMedQA, and BioASQ.
desk verdict Plausible fine-tuning results, but the RAG evaluation is circular and the splits are missing, so the headline numbers aren't verifiable. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the fine-tuned model itself, MedBioLM, built by supervised fine-tuning of GPT-4o on task-specific datasets (MedQA, PubMedQA, BioASQ, MedicationQA, LiveQA, and a custom combined set) with prompt-specific decoding parameters. The fine-tuning pipeline is the mechanism: adaptive optimization with batch size set to 0.2% of training examples, a dynamic learning-rate scaling factor, and automatic seed assignment. RAG is implemented as a keyword-based search index with document chunking and structured field mapping, but the experiments show that fine-tuning, not retrieval, carries the accuracy gains. Prompt engineering (system messages, max tokens, temperature, top-p, penalties) is tuned per QA format, and evaluation uses accuracy for closed-ended tasks and ROUGE, BLEU, BERTScore, and BLEURT for open-ended ones.
What would settle it
Run MedBioLM's exact evaluation on the official held-out test splits of MedQA, PubMedQA, and BioASQ, with training sets explicitly excluded, and check whether the accuracy margins over base GPT-4o (88.0 vs 87.0, 78.9 vs 44.74, 96.0 vs 92.0) and the ROUGE/BLEU gains still appear; the claim collapses if the margins vanish.
Extended reading notes
Core claim
The central discovery is that supervised fine-tuning of GPT-4o on medical and biological QA datasets produces a model—MedBioLM—that substantially outperforms the original base model on closed-ended reasoning tasks (MedQA 88.0% vs 87.0%, PubMedQA 78.9% vs 44.74%, BioASQ 96.0% vs 92.0%) and improves long-form and short-form answer quality as measured by ROUGE, BLEU, BERTScore, and BLEURT. The paper also finds that RAG provides only marginal gains for the base model on short-form QA and no significant gain once fine-tuning is applied, and that increasing retrieval top-k beyond one hurts short-form answer quality. Fine-tuning reduces the model's tendency to answer 'Maybe' on PubMedQA, indicating improved confidence and reasoning. The authors present this as evidence that domain-specific adaptation is the primary driver of biomedical QA performance, with retrieval playing a secondary role.
Load-bearing premise
The reported gains assume the benchmark questions were not seen during fine-tuning, yet no train/test split is stated anywhere in the paper.
Editorial extensions
If this is right
- Domain-specific fine-tuning of a strong general LLM can lift medical exam performance by 10–30 percentage points over the base model.
- Retrieval-augmented generation built on keyword search gives only marginal short-form QA gains and can hurt performance as top-k grows.
- Fine-tuned medical QA models answer decisively instead of hedging with 'Maybe', which matters for clinical decision support.
- Combining multiple QA datasets during fine-tuning produces the best long-form scores, implying data diversity helps generalization.
Reading between the lines
- If the reported gains hold on truly held-out splits, the implication is that fine-tuning GPT-4o is a cheap, practical route to specialized medical assistants without building a new model from scratch.
- The negative top-k result hints that for short medical answers, precision-oriented retrieval with a single best document outperforms broad recall—a design principle that could generalize to other closed-domain QA settings.
- The single-physician pairwise evaluation leaves open whether human readers would actually prefer MedBioLM's long answers; a multi-expert blind study would be the natural next test.
- The paper does not report train/test splits, so a clean re-evaluation with explicitly disjoint data would separate genuine domain adaptation from memorization of benchmark questions.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces MedBioLM, a GPT-4o model fine-tuned on several biomedical QA datasets, optionally combined with a keyword-based retrieval-augmented generation (RAG) pipeline. The authors report accuracy gains on closed-ended benchmarks (MedQA 88.0%, PubMedQA 78.9%, BioASQ 96.0%) and improvements in ROUGE, BLEU, BERTScore, and BLEURT on long-form and short-form QA, and they conclude that fine-tuning is the dominant contributor while RAG provides inconsistent or marginal gains. The manuscript also includes training configurations, prompting strategies, and an analysis of top-k retrieval behavior.
Significance. If the reported numbers were valid, the paper would offer a useful practical recipe for adapting a general-purpose LLM to biomedical QA across multiple answer formats, and its negative results on RAG would be informative. The study addresses a relevant and active problem, and the authors appropriately acknowledge limitations such as single-physician evaluation and overfitting risk. However, the empirical evaluation as presented is not verifiable: the paper does not specify train/test splits, the RAG index stores gold answers, and several reported numbers contradict each other. These issues go beyond presentation and affect the central claims, so the current contribution cannot be assessed on the evidence provided.
major comments (5)
- [§3.1, Tables 1, 3, 4, 5] The paper never states whether any evaluation was performed on held-out test splits. Table 5 enumerates only training samples and durations, and §3.1 describes the combined custom dataset as "curated for training," yet Table 3 reports MedBioLM's long-form scores on exactly that dataset. Without stated test splits, the numbers in Tables 1, 3, and 4 cannot be distinguished from memorization of training targets, which invalidates the abstract's claim that fine-tuning "significantly improves" performance.
- [§3.3, Table 4, Figure 4] The retrieval index stores an answer field containing "corresponding responses," and both question and answer fields are searchable. For any evaluation question present in the index, keyword retrieval can return the gold answer, and the generator can then copy it. Consequently, the MedBioLM+RAG rows in Table 4 and the top-k analysis in Figure 4 do not measure retrieval of external knowledge; they measure copying from the answer key. The manuscript must state that test queries and their gold answers were excluded from the index, and the methodology must actually implement that exclusion.
- [§4.3, Table 4, Figure 4] The reported numbers are internally inconsistent. Table 4 gives MedBioLM+RAG ROUGE-1=40.51 and BLEU=11.55, while §4.3 reports top-k=1 values of ROUGE-1=11.33 and BLEU=3.26 for the same short-form MedQA setup; either Figure 4 refers to a different configuration or one of the reported values is wrong. In addition, §4.3 states that "the fine-tuned GPT-4o model achieves a BLEURT score of 8.63," whereas Table 4 lists MedBioLM BLEURT=33.43 and MedBioLM+RAG BLEURT=8.63. The text and the table cannot both be correct, making the central RAG comparison unverifiable.
- [§4, Tables 1, 3, 4] No error bars, confidence intervals, significance tests, or multiple training runs are reported. For example, Table 1 shows MedBioLM at 88.0% versus GPT-4o at 87.0% on MedQA; a one-point difference with no variance or significance assessment does not support the claim of "significant" improvement. The same issue applies to the small ROUGE/BLEU differences in Tables 3 and 4, where the relative gains are on the order of a few points or less.
- [§3.1, §5, Table 5] The "Combined Custom Dataset" used to train and evaluate MedBioLM in Table 3 is not described in sufficient detail and is not released. The text says it "consists of a combination of long-form QA data from LiveQA, MedQA, PubMedQA, and BioASQ," but Table 5 lists 6,652 samples without explaining how the constituent datasets were filtered, split, or balanced. Without a precise description and release, the Table 3 results are not reproducible, particularly because the same dataset is described as being "curated for training."
minor comments (6)
- [Table 2] Table 2 has broken column headers such as "Y es (%)" and row entries that run together (e.g., "78.9 64 36 96"); the table needs to be reformatted.
- [Throughout] The dataset name is inconsistently spelled as both "BioASQ" and "BioSQA" in the text and tables; please unify the terminology.
- [§4.3] The sentence beginning "The short-form question evaluation results indicate..." is repeated verbatim in consecutive paragraphs and should be removed.
- [§3.4] There is a typo: "expolored" should be "explored."
- [Figure 4] Figure 4's axes and configuration are not defined clearly; the text reports specific k=1 and k=5 values but does not say whether these correspond to the base model or the fine-tuned model, and the values do not match Table 4.
- [§4.2] The pairwise evaluation by a single physician is acknowledged as subjective, but the number of questions evaluated and the selection criteria are not stated; please provide this information.
Circularity Check
Evaluation design makes the reported gains partly self-produced: the custom 'training' dataset is used as the MedBioLM evaluation set, and the RAG index stores the answer field so retrieval can supply the gold answer.
-
fitted input called prediction
[Section 3.1 (Datasets), Table 3 (Long-form QA), Table 5 (Fine-tuning details)]
"We also created a custom dataset curated for training biomedical-specific language models, incorporating various QA formats from multiple sources to improve medical reasoning and factual accuracy. Table 5 lists 'Combined Custom Dataset 6,652' under Train Dataset; Table 3 reports 'MedBioLM Custom Dataset 26.67 8.72 18.71 3.12 12.08 -30.26'."
The custom dataset is explicitly described as 'curated for training' and appears in Table 5 as the training set. Table 3 then reports MedBioLM's long-form ROUGE, BLEU, BERTScore, and BLEURT on that same 'Custom Dataset' with no stated held-out split. The reported scores therefore measure the model's fit to its own training targets rather than generalization, so the 'prediction' is the training fit relabeled as an evaluation result.
-
self definitional
[Section 3.3 (Retrieval-Augmented Generation), Section 4.3 (Short-form QA), Table 4]
"The indexed fields included an ID field as a unique identifier, a question field for medical queries, and an answer field containing corresponding responses. The ID field was designated as the primary key to maintain data integrity, while both the question and answer fields were configured as searchable and retrievable."
The RAG index stores the answer field and explicitly makes it searchable and retrievable. For evaluation questions that appear in the index, keyword retrieval can return the gold reference answer, which is then fed to the generator. Table 4 and Figure 4 compare MedBioLM with and without RAG on MedQA short-form, so the RAG rows measure how well the model copies from the answer key, not how well retrieval supplies external knowledge. The RAG evaluation is therefore circular by construction.
full rationale
The paper's two headline contributions, fine-tuning gains and RAG-based factual consistency, are both evaluated in settings that can feed the ground truth into the reported numbers. Section 3.1 describes the custom dataset as 'curated for training,' Table 5 lists it as the training set, and Table 3 reports MedBioLM's long-form scores on that same dataset without any stated train/test split, so those scores can reflect memorization. Separately, Section 3.3 defines the retrieval index with a searchable answer field, so for any indexed evaluation question the RAG pipeline can return the gold answer to the generator; Table 4 and Figure 4 therefore do not test external-knowledge retrieval. The closed-ended benchmark comparisons in Table 1 are not inherently circular, because standard datasets often have official splits, but the paper never states the splits for any dataset, so they cannot compensate for the two explicit reductions. Together these issues make the central evaluation partially circular, corresponding to a score of 7 rather than a fully forced derivation.
Assumptions & free parameters
assumptions (4)
- domain assumption The benchmark datasets (MedQA, PubMedQA, BioASQ) and their standard evaluation protocols are reliable measures of biomedical QA quality.
- ad hoc to paper Evaluation was performed on held-out test splits of the training datasets.
- ad hoc to paper The RAG index provides external knowledge rather than the gold answer for the query.
- domain assumption A single physician's pairwise evaluation is representative of clinical quality.
Cite this review
Pith. "Pith review of MedBioLM: Optimizing Medical and Biological QA with Fine-Tuned Large Language Models and Retrieval-Augmented Generation." pith.science (2026). https://pith.science/paper/3VHDQYGD
@misc{pith2026250203004,
author = {Pith},
title = {Pith review of: MedBioLM: Optimizing Medical and Biological QA with Fine-Tuned Large Language Models and Retrieval-Augmented Generation},
year = {2026},
howpublished = {\url{https://pith.science/paper/3VHDQYGD}},
note = {Machine review of arXiv:2502.03004}
}
read the original abstract
Large Language Models (LLMs) have demonstrated impressive capabilities across natural language processing tasks. However, their application to specialized domains such as medicine and biology requires further optimization to ensure factual accuracy, reliability, and contextual depth. We introduce MedBioLM, a domain-adapted biomedical question-answering model designed to enhance both short-form and long-form queries. By integrating fine-tuning and retrieval-augmented generation (RAG), MedBioLM dynamically incorporates domain-specific knowledge, improving reasoning abilities and factual accuracy. To evaluate its effectiveness, we fine-tuned the model on diverse biomedical QA datasets, covering structured multiple-choice assessments and complex clinical reasoning tasks. Fine-tuning significantly improves accuracy on benchmark datasets, while RAG enhances factual consistency. These results highlight the potential of domain-optimized LLMs in advancing biomedical research, medical education, and clinical decision support.
Figures
Figures from the paper (3 more)
Forward citations
Cited by 3 Pith papers
-
When Retrieval Helps and Distracts: Evaluating Evidence-Generating LLMs for Biomedical Claim Verification
On the five-source CARE-XAI benchmark, classifiers beat LLMs at verdict-only prediction, fine-tuning improves evidence-generating LLMs more than retrieval, and PubMed retrieval helps only source-aligned claims.
-
Aligning Clinical Needs and AI Capabilities: A Survey on LLMs for Medical Reasoning
A dual clinical-computational taxonomy for medical LLM reasoning plus a five-level 5k-sample benchmark showing specialists excel at diagnosis and general models at decision support/dialogue.
-
AfriEconQA: A Benchmark for Quantitative and Temporal Reasoning over World Bank Economic Reports
An LLM-generated QA benchmark over World Bank African economic reports is evaluated on a small sample and found hard, but the abstract and body report conflicting dataset sizes, model names, and scores.
Reference graph
Works this paper leans on
-
[1]
Abacha, A. B. and Demner-Fushman, D. A question-entailment approach to question answering. BMC Bioinformatics, 20 0 (1): 0 511:1--511:23, 2019. doi:10.1186/s12859-019-3119-4. URL https://bmcbioinformatics.biomedcentral.com/articles/10.1186/s12859-019-3119-4
-
[2]
B., Agichtein, E., Pinter, Y., and Demner-Fushman, D
Abacha, A. B., Agichtein, E., Pinter, Y., and Demner-Fushman, D. Overview of the medical question answering task at TREC 2017 LiveQA . In Proceedings of the Text REtrieval Conference (TREC) 2017. National Institute of Standards and Technology (NIST), 2017
work page 2017
-
[3]
B., Mrabet, Y., Sharp, M., Goodwin, T., Shooshan, S
Abacha, A. B., Mrabet, Y., Sharp, M., Goodwin, T., Shooshan, S. E., and Demner-Fushman, D. Bridging the gap between consumers' medication questions and trusted answers. In Proceedings of the 17th World Congress on Medical and Health Informatics ( MEDINFO 2019) , volume 264, pp.\ 25--29. IOS Press, 2019. doi:10.3233/SHTI190176. URL https://ebooks.iospress....
-
[4]
Chen, Z., Cano, A. H., Romanou, A., Bonnet, A., Matoba, K., Salvi, F., Pagliardini, M., Fan, S., Köpf, A., Mohtashami, A., Sallinen, A., Sakhaeirad, A., Swamy, V., Krawczuk, I., Bayazit, D., Marmet, A., Montariol, S., Hartley, M.-A., Jaggi, M., and Bosselut, A. Meditron-70b: Scaling medical pretraining for large language models, 2023. URL https://arxiv.or...
arXiv 2023
-
[5]
Olaph: Improving factuality in biomedical long-form question answering, 2024
Jeong, M., Hwang, H., Yoon, C., Lee, T., and Kang, J. Olaph: Improving factuality in biomedical long-form question answering, 2024. URL https://arxiv.org/abs/2405.12701
arXiv 2024
-
[6]
Jin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., and Szolovits, P. What disease does this patient have? a large-scale open domain question answering dataset from medical exams, 2020. URL https://arxiv.org/abs/2009.13081
arXiv 2020
-
[7]
Pubmedqa: A dataset for biomedical research question answering
Jin, Q., Dhingra, B., Liu, Z., Cohen, W., and Lu, X. Pubmedqa: A dataset for biomedical research question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing ( EMNLP-IJCNLP ) , pp.\ 2567--2577. Association for Computational Linguistic...
-
[8]
Retrieval-augmented generation for knowledge-intensive nlp tasks
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., tau Yih, W., Rocktäschel, T., Riedel, S., and Kiela, D. Retrieval-augmented generation for knowledge-intensive nlp tasks. In Advances in Neural Information Processing Systems (NeurIPS 2020), volume 33, pp.\ 9459--9474. Curran Associates, Inc., 2020. URL https:...
work page 2020
Show all 24 references
-
[9]
Biogpt: generative pre-trained transformer for biomedical text generation and mining
Luo, R., Sun, L., Xia, Y., Qin, T., Zhang, S., Poon, H., and Liu, T.-Y. Biogpt: generative pre-trained transformer for biomedical text generation and mining. Briefings in Bioinformatics, 23 0 (6), September 2022. ISSN 1477-4054. doi:10.1093/bib/bbac409. URL http://dx.doi.org/1...
2022 doi
-
[10]
S., Prakash, S., Pathak, A., Semturs, C., Patel, S., Webster, D
McDuff, D., Schaekermann, M., Tu, T., Palepu, A., Wang, A., Garrison, J., Singhal, K., Sharma, Y., Azizi, S., Kulkarni, K., Hou, L., Cheng, Y., Liu, Y., Mahdavi, S. S., Prakash, S., Pathak, A., Semturs, C., Patel, S., Webster, D. R., Dominowska, E., Gottweis, J., Barral, J., C...
2023 arXiv
-
[11]
Azure AI Search , 2024 a
Microsoft Azure . Azure AI Search , 2024 a . URL https://learn.microsoft.com/en-us/azure/search/search-what-is-azure-search. Accessed: 2024-01-31
2024
-
[12]
Fine-tune models with Azure AI Foundry , 2024 b
Microsoft Azure . Fine-tune models with Azure AI Foundry , 2024 b . URL https://learn.microsoft.com/en-us/azure/ai-studio/concepts/fine-tuning-overview. Accessed: 2024-01-31
2024
-
[13]
Overview of BioASQ 2023: The Eleventh BioASQ Challenge on Large-Scale Biomedical Semantic Indexing and Question Answering, pp.\ 227–250
Nentidis, A., Katsimpras, G., Krithara, A., Lima López, S., Farré-Maduell, E., Gasco, L., Krallinger, M., and Paliouras, G. Overview of BioASQ 2023: The Eleventh BioASQ Challenge on Large-Scale Biomedical Semantic Indexing and Question Answering, pp.\ 227–250. Springer Nature ...
2023 doi
-
[14]
T., Zhang, S., Carignan, D., Edgar, R., Fusi, N., King, N., Larson, J., Li, Y., Liu, W., Luo, R., McKinney, S
Nori, H., Lee, Y. T., Zhang, S., Carignan, D., Edgar, R., Fusi, N., King, N., Larson, J., Li, Y., Liu, W., Luo, R., McKinney, S. M., Ness, R. O., Poon, H., Qin, T., Usuyama, N., White, C., and Horvitz, E. Can generalist foundation models outcompete special-purpose tuning? case...
2023 arXiv
-
[15]
Ecg-qa: A comprehensive question answering dataset combined with electrocardiogram
Oh, J., Lee, G., Bae, S., myoung Kwon, J., and Choi, E. Ecg-qa: A comprehensive question answering dataset combined with electrocardiogram. In Advances in Neural Information Processing Systems (NeurIPS 2023), volume 36, pp.\ 66277--66288. Curran Associates, Inc., 2023
2023
-
[16]
Gpt-4o system card, 2024 a
OpenAI. Gpt-4o system card, 2024 a . URL https://arxiv.org/abs/2410.21276
2024 arXiv
-
[17]
Gpt-4 technical report, 2024 b
OpenAI. Gpt-4 technical report, 2024 b . URL https://arxiv.org/abs/2303.08774
2024 arXiv
-
[18]
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P., Leike, J., and Lowe, R. Training language models to f...
2022 arXiv
-
[19]
Z., Hu, S.-Y., Schaekermann, M., Kamath, A., Cheng, Y., Barrett, D
Saab, K., Tu, T., Weng, W.-H., Tanno, R., Stutz, D., Wulczyn, E., Zhang, F., Strother, T., Park, C., Vedadi, E., Chaves, J. Z., Hu, S.-Y., Schaekermann, M., Kamath, A., Cheng, Y., Barrett, D. G. T., Cheung, C., Mustafa, B., Palepu, A., McDuff, D., Hou, L., Golany, T., Liu, L.,...
2024 arXiv
-
[20]
S., Wei, J., Chung, H
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., Payne, P., Seneviratne, M., Gamble, P., Kelly, C., Scharli, N., Chowdhery, A., Mansfield, P., y Arcas, B. A., Webster, D., Corrado, G. S., Matias, Y., Cho...
2022 arXiv
-
[21]
S., Wei, J., Chung, H
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., Payne, P., Seneviratne, M., Gamble, P., Kelly, C., Babiker, A., Schärli, N., Chowdhery, A., Mansfield, P., Demner-Fushman, D., Agüera y Arcas, B., Webster...
2023
-
[22]
and Gómez-Rodríguez, C
Vilares, D. and Gómez-Rodríguez, C. Head-qa: A healthcare dataset for complex reasoning. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL 2019), pp.\ 960--966, Florence, Italy, 2019. Association for Computational Linguistics. doi:...
2019 doi
-
[23]
D., Ren, H., Huang, J., Chen, C., Zhou, Y., Fu, S., Liu, W., Liu, T., Li, X., Chen, Y., He, L., Zou, J., Li, Q., Liu, H., and Sun, L
Zhang, K., Zhou, R., Adhikarla, E., Yan, Z., Liu, Y., Yu, J., Liu, Z., Chen, X., Davison, B. D., Ren, H., Huang, J., Chen, C., Zhou, Y., Fu, S., Liu, W., Liu, T., Li, X., Chen, Y., He, L., Zou, J., Li, Q., Liu, H., and Sun, L. A generalist vision–language foundation model for ...
2024 doi
-
[24]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...
Reviewed August 9, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.