REVIEW 4 major objections 5 minor 14 references
Open-Source Retrieval Augmented Generation Framework for Retrieving Accurate Medication Insights from Formularies for African Healthcare Workers
T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read An open-source RAG chatbot built on the Nigerian EMDEX formulary returns medication answers that match pharmacist answers at 84% average sentence similarity, supporting faster drug lookups for African healthcare workers.
desk verdict A useful open-source RAG pilot for Nigerian formularies, but the evaluation metric under-sells the hallucination claim. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the retrieval-augmented generation loop together with the S-BERT evaluation metric. Sentence-BERT (S-BERT) is a sentence-embedding model that produces a cosine similarity score between two texts, and the paper uses that score to compare Drug Insights' answers with a pharmacist's answers to the same queries. The pipeline itself works through layout-aware text extraction from EMDEX PDFs, schema-based normalization of drug entries by a language model, 1536-dimensional embedding and vector-database storage with top-3 retrieval at a 0.9 cosine threshold, and final answer generation by a GPT-based model from the retrieved passages. Prompt engineering supplies the controlled variation: nine zero-shot prompts differing in output-sentence limits and in whether they use comparison or guardrail instructions, with the guardrail-plus-no-limit combination winning.
What would settle it
Have an independent clinical panel score each chatbot answer against the full EMDEX formulary and standard references, flagging factual errors, omissions, and unsupported statements. If answers that scored high on semantic similarity to a pharmacist's short answer still contain material factual errors—especially on questions the formulary cannot answer—then the reported 84% similarity score would not establish accurate, low-hallucination drug responses.
Extended reading notes
Core claim
The central claim is that a corpus-bound RAG system can serve as a reliable drug-information lookup for African healthcare workers: given a question, Drug Insights retrieves relevant passages from the EMDEX formulary and generates an answer that, on average, is semantically close to what a pharmacist would write using only that same corpus. The quantitative anchor is the 84.09% average S-BERT cosine similarity achieved by prompt_0a, which pairs guardrail instructions with no sentence limit; the lowest-scoring prompt, prompt_1b, reached 65.91%. Human evaluators from a healthcare-technology partner rated the outputs between 3.76 and 3.88 on a five-point scale for accuracy, relevance, usefulness, and source relevance. The authors interpret these results as evidence that RAG with prompt engineering can reduce hallucination and provide context-limited, trustworthy medication answers.
Load-bearing premise
The whole accuracy claim rests on treating the similarity score between the chatbot's answer and a pharmacist's answer—where the pharmacist could only use the same drug handbook—as proof that the answer is clinically correct and free of made-up information; that equivalence is assumed, not checked against an independent source of truth.
Editorial extensions
If this is right
- If the 84.09% similarity result holds, the best prompt can produce formulary answers semantically close to a pharmacist's short answer, making the tool usable as a first-pass drug-information resource where pharmacist time is scarce.
- Because the framework is open source, other organizations can substitute their own formulary PDFs and reuse the extraction-to-evaluation pipeline without rebuilding it.
- The spread from 65.91% to 84.09% across prompt variants implies that prompt design is a primary determinant of RAG answer quality in this setting.
- The pharmacist agreement scores around 3.8/5 support using the tool as a first-pass reference, while the paper's stated future work shows it is not yet positioned as a substitute for professional clinical review.
Reading between the lines
- The similarity score measures agreement with a pharmacist who was restricted to the same corpus, so a high score chiefly shows the chatbot can retrieve and restate what a careful reader would find in EMDEX; independent clinical validation would be needed to show the facts themselves are correct.
- The paper includes a few questions whose answers are not in the corpus but does not report their scores, so the most direct untested question is whether the system declines to answer or silently fabricates when coverage fails.
- The fixed 0.9 cosine threshold and top-3 retrieval are fixed design choices; varying them would separate retrieval quality from generation style as drivers of the 84.09% score.
- The evaluation is limited to Nigerian formulary content, so extending the same pipeline to another country's formulary is a natural test of whether the framework generalizes across African healthcare settings.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents 'Drug Insights,' an open-source retrieval-augmented generation (RAG) chatbot that answers medication queries from a corpus of Nigerian pharmaceutical data, specifically the EMDEX formulary. The system uses Azure OpenAI embeddings, a Pinecone vector database, and GPT-4o for answer generation, with nine prompt variants compared via S-BERT similarity against pharmacist-written answers to 50 queries. The authors report that the best prompt (prompt_0a) achieves an average S-BERT similarity of 84.09%, and they supplement this with pharmacist Likert-scale feedback (averages 3.76–3.88 out of 5). The conclusion claims the system 'delivers accurate, context-specific responses with minimal hallucination.' The paper also describes future plans for retrieval precision/recall evaluation, chat-session testing, and additional similarity metrics.
Significance. The problem addressed is real and consequential: African healthcare workers need rapid, reliable access to medication information, and an open-source, locally relevant RAG tool could help. The authors should be credited for building and releasing the system, grounding it in a real formularies corpus, involving pharmacists in question design and evaluation, and systematically comparing multiple prompt variants. If the accuracy and hallucination claims were rigorously validated, this would be a useful contribution to applied medical NLP and information retrieval. However, the current evaluation does not substantiate those claims: the primary metric (S-BERT similarity to pharmacist answers derived from the same corpus) is not a measure of clinical correctness, the prompt selected as best was chosen on the same test set used to report its score, and the paper itself acknowledges that retrieval recall/precision was not evaluated. The significance is therefore contingent on substantial additional validation.
major comments (4)
- [Section III.B] The central evaluation compares Drug Insights responses with pharmacist answers produced using the same EMDEX corpus and reports S-BERT cosine similarity as the evidence for 'accurate, context-specific responses with minimal hallucination.' This is not a measure of clinical accuracy or hallucination: a semantically similar answer can contain a wrong dosage or contraindication, and a semantically dissimilar answer can be clinically correct. No independent gold standard, sentence-level fact checking, or inter-rater reliability of the pharmacist reference answers is provided. The paper's own Section V lists 'Explicit systematic evaluation of the recall and precision of database retrievals' as future work, which further weakens the current claim.
- [Section III.B and Fig. 2] The best prompt, prompt_0a, was selected as the one with the highest average S-BERT similarity on the same 50-query set used to report the 84.09% score. This makes the reported number a test-set maximum rather than a held-out estimate, inflating apparent quality. The authors should either split the data into prompt-selection and evaluation sets, use cross-validation, or at minimum report confidence intervals and the selection procedure. Without this, the comparison across prompts in Fig. 2 cannot support any claim about which prompt is best.
- [Section II.C] The retrieval configuration (cosine similarity threshold of 0.9, similarity count of 3) is presented as a fixed design choice without any sensitivity analysis. These are free parameters that can substantially affect retrieval quality, and therefore the final answer. The paper should include a brief sensitivity study or at least a rationale grounded in experiments, especially because Section V acknowledges that retrieval recall/precision has not been evaluated.
- [Section III.B] The query set of 50 questions is small, and the paper states that 'a few questions whose answers were not present in the corpus' were included as a hallucination check, but the results for those questions are not reported separately. This is a missed opportunity: reporting how the system handled out-of-corpus queries would speak directly to the hallucination claim. At minimum, the authors should quantify how many such questions were included and show their individual scores or a qualitative assessment.
minor comments (5)
- [Section IV heading] The heading 'IV. PREPARE HOW IT WORKS, INCLUDING ARCHITECTURE AND PERFORMANCE METRICS' appears garbled; this likely should be 'SYSTEM OVERVIEW' or 'HOW IT WORKS.'
- [Section II.A] The sentence 'Maintaining the Integrity of the Specifications' appears at the end of Section II.A and is unrelated to the surrounding text; it appears to be leftover boilerplate and should be removed.
- [Figures] The paper refers to 'Fig 2' when describing the average S-BERT similarity scores, but the figure caption in the text says 'Fig 2' and the later figure is numbered 'Fig 3' while the text refers to 'Fig 3' for the sample query image. The figure numbering and in-text references should be re-checked; also, the sample query image in Fig. 3 is not discussed in the body.
- [Table 2] The pharmacist feedback table reports averages but not the number of pharmacists who rated the responses, the number of queries each rated, or the standard deviation. Adding these details would make the feedback more interpretable.
- [General] The abstract claims the system 'delivers accurate, context-specific responses with minimal hallucination,' but the body results are characterized as 'Preliminary tests' (Abstract) and the conclusion uses 'validated by S-BERT metrics and pharmacist feedback.' The claims should be moderated to match the evidence, e.g., 'shows promise' rather than 'delivers.'
Circularity Check
No significant circularity: the S-BERT comparison is a descriptive evaluation and the prompt selection is not presented as a held-out prediction.
full rationale
This is an engineering report rather than a derivation, so most circularity patterns do not apply. The central quantitative result, the 84.09% S-BERT similarity score for prompt_0a, is presented as the outcome of comparing nine prompts on a 50-query test set, not as a prediction of future performance. Selecting the best prompt on the same set is a model-selection procedure, and the paper does not call the chosen score a held-out estimate. The pharmacist reference answers were produced using the same corpus available to Drug Insights, which makes the S-BERT metric a consistency check rather than an independent clinical-accuracy measurement; this is a validity limitation of the evaluation, not a circular derivation, because the score is not defined in terms of the system's own outputs or fitted to guarantee the claimed accuracy. The paper also explicitly acknowledges that retrieval precision and recall were not systematically evaluated, further confirming that no fitted retrieval parameter is being renamed as a prediction. References to Advantage Health Africa and EMDEX are contextual or evaluative, not load-bearing self-citations that force the paper's conclusions. Overall, the accuracy and minimal-hallucination claims are under-supported by the evidence, but no step in the paper reduces by construction to its own inputs.
Assumptions & free parameters
free parameters (3)
- cosine similarity threshold =
0.9
- similarity count (top-k) =
3
- prompt_0a selection =
best prompt
assumptions (3)
- domain assumption EMDEX formulary is an accurate and comprehensive reference for Nigerian medications
- domain assumption Pharmacist responses using the same corpus are a valid gold standard for correct answers
- domain assumption S-BERT cosine similarity correlates with clinical answer quality
Cite this review
Pith. "Pith review of Open-Source Retrieval Augmented Generation Framework for Retrieving Accurate Medication Insights from Formularies for African Healthcare Workers." pith.science (2026). https://pith.science/paper/54MDUSPX
@misc{pith2026250215722,
author = {Pith},
title = {Pith review of: Open-Source Retrieval Augmented Generation Framework for Retrieving Accurate Medication Insights from Formularies for African Healthcare Workers},
year = {2026},
howpublished = {\url{https://pith.science/paper/54MDUSPX}},
note = {Machine review of arXiv:2502.15722}
}
read the original abstract
Accessing accurate medication insights is vital for enhancing patient safety, minimizing errors, and supporting clinical decision-making. However, healthcare professionals in Africa often rely on manual and time-consuming processes to retrieve drug information, exacerbated by limited access to pharmacists due to brain drain and healthcare disparities. This paper presents "Drug Insights," an open-source Retrieval-Augmented Generation (RAG) chatbot designed to streamline medication lookup for healthcare workers in Africa. By leveraging a corpus of Nigerian pharmaceutical data and advanced AI technologies, including Pinecone databases and GPT models, the system delivers accurate, context-specific responses with minimal hallucination. The chatbot integrates prompt engineering and S-BERT evaluation to optimize retrieval and response generation. Preliminary tests, including pharmacist feedback, affirm the tool's potential to improve drug information access while highlighting areas for enhancement, such as UI/UX refinement and extended corpus integration.
Figures
Reference graph
Works this paper leans on
-
[1]
Qadus S, Naser AY, Al -Rousan R, Daghash A. Utilization of drug information resources among community pharmacists in Jordan: A cross-sectional study. Saudi Pharm J. 2022 Jan;30(1):1 -7. doi: 10.1016/j.jsps.2021.12.001. Epub 2021 Dec 8. PMID: 35145341; PMCID: PMC8802090
-
[2]
Utilization of drug information services in selected tertiary hospitals in Enugu state, Nigeria
Ogbonna B and Okoye IM. Utilization of drug information services in selected tertiary hospitals in Enugu state, Nigeria. Journal of Current Biomedical Research, 2021. https://journals.unizik.edu.ng/jcbr/article/download/718/678/1852
work page 2021
-
[3]
Yusuf, Hadiza & Abdu-Aguye, Samirah & Suleiman, Habeeb & MH, Modi & Labaran, Kamilu & Abdulmalik, Ahmed. (2020). Utilization of Electronic Drug Information Resources by Pharmacists Practicing in Hospital and Community Settings in Maiduguri, Nigeria. 16. 3138- 3145
work page 2020
-
[4]
M. Maryamah, M. M. Irfani, E. B. Tri Raharjo, N. A. Rahmi, M. Ghani, and I. K. Raharjana, "Chatbots in Academia: A Retrieval - Augmented Generation Approach for Improved Efficient Information Access," in 2024 16th International Conference on Knowledge and Smart Technology (KST), pp. 259 –264, IEEE, 2024. [Online]. Available: https://doi.org/10.1109/KST612...
-
[5]
Thüs, D., Malone, S., & Brünken, R. (2024). Exploring generative AI in higher education: a RAG system to enhance student engagement with scientific literature. Frontiers in Psychology, 15, 1474892
work page 2024
-
[6]
C. Jeong, "A Study on the Implementation of Generative AI Services Using an Enterprise Data -Based LLM Application Architecture," Advance online publication, 2023. [Online]. Available: https://doi.org/10.48550/arXiv.2309.01105
work page Pith review arXiv doi:10.48550/arxiv.2309.01105 2023
-
[7]
Ahmad, Syed Rameel. "Enhancing Multilingual Information Retrieval in Mixed Human Resources Environments: A RAG Model Implementation for Multicultural Enterprise." arXiv preprint arXiv:2401.01511 (2024)
arXiv 2024
-
[8]
Opportunities and challenges for ChatGPT and large language models in biomedicine and health,
S. Tian, Q. Jin, L. Yeganova, P.-T. Lai, Q. Zhu, X. Chen, Y. Yang, Q. Chen, W. Kim, D. C. Comeau, et al., "Opportunities and challenges for ChatGPT and large language models in biomedicine and health," Briefings in Bioinformatics, vol. 25, p. bbad493, 2024
work page 2024
Show all 14 references
-
[9]
BioGPT: Generative pre -trained transformer for biomedical text generation and mining,
R. Luo, L. Sun, Y. Xia, T. Qin, S. Zhang, H. Poon, and T. -Y. Liu, "BioGPT: Generative pre -trained transformer for biomedical text generation and mining," Briefings in Bioinformatics, vol. 23, p. bbac409, 2022. [Online]. Available: https://doi.org/10.1093/bib/bbac409
2022 doi
-
[10]
EMDEX: The Essential Medicines Index,
"EMDEX: The Essential Medicines Index," [Online]. Available: https://emdex.org/about/
-
[11]
Prompt Artisan,
A. Alvares, "Prompt Artisan," [Online]. Available: https://chatgpt.com/g/g-tH8fLNSDw-prompt-artisan
-
[12]
The Prompt Report: A Systematic Survey of Prompting Techniques,
S. Schulhoff, et al., "The Prompt Report: A Systematic Survey of Prompting Techniques," 2024
2024
-
[13]
Sentence -BERT: Sentence Embeddings using Siamese BERT-Networks,
N. Reimers, et al., "Sentence -BERT: Sentence Embeddings using Siamese BERT-Networks," 2019
2019
-
[14]
Advantage Health Africa,
"Advantage Health Africa," [Online]. Available: https://advantagehealthafrica.com/
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.