{"id":"537ba40a-8899-4f1c-9eaf-2769291d83af","arxiv_id":"2502.15722","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Drug Insights, an open-source RAG chatbot over the Nigerian EMDEX formulary, achieved 84% S-BERT similarity to pharmacist answers in a 50-query test, but the evaluation method is circular and lacks baselines.","lead":"Researchers built Drug Insights, an open-source chatbot that answers medication questions for African healthcare workers using a Nigerian drug formulary. Early tests suggest it can retrieve relevant drug information, but the evaluation is weak and the accuracy claims are not strongly supported.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"S-BERT similarity to a pharmacist's answer does not measure clinical accuracy or hallucination, so the 84.09% score does not substantiate the abstract's 'accurate, context-specific responses with minimal hallucination' claim.","rationale":"The reader's weakest assumption correctly identifies the load-bearing issue: S-BERT similarity to a pharmacist answer is not a validated measure of clinical accuracy or hallucination. My stress-test agrees with this and adds that prompt selection on the same test set further weakens the quantitative headline. The system itself is plausible and the authors are transparent about limitations in Section V, so a conditional verdict remains appropriate. No reason to change the reader's verdict; the requested revisions should include a clinical fact-level evaluation and a held-out prompt evaluation before the accuracy claim can be taken as demonstrated.","tokens_in":4710,"tokens_out":2449,"duration_ms":25133,"concrete_test":"Construct a gold-standard answer key for the 50 queries by having two independent pharmacists, blinded to system outputs, extract the exact drug facts (dose, indication, contraindication, side effect) from the EMDEX corpus. Score each Drug Insights response at the statement level for factual precision/recall against this key, and count unsupported statements as hallucinations. Then fix prompt_0a and evaluate it on a new held-out set of 50 queries not used for prompt selection. If statement-level accuracy is not measured, or if the held-out accuracy differs materially from the 84.09% S-BERT score, the central accuracy/minimal-hallucination claim is not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim rests on Section III.B, where the Drug Insights response is compared to a pharmacist's response using S-BERT cosine similarity. The implicit assumption is that high semantic similarity to a pharmacist's answer, who was restricted to the same EMDEX corpus, is a valid proxy for clinical accuracy and absence of hallucination. This assumption is not established. A response can be semantically similar while containing a clinically wrong dosage or contraindication, or it can be semantically dissimilar simply because the pharmacist used a different phrase or level of detail. The evaluation also has no independent gold standard, no statement-level fact checking, and no inter-rater reliability for the pharmacist reference answers. In addition, prompt_0a was selected as the best prompt using the same 50-query test set, so the reported 84.09% is a test-set maximum rather than a held-out estimate; this inflates the apparent quality. The paper itself acknowledges in Section V that explicit recall/precision of retrieval was not evaluated. Therefore the abstract's accuracy and minimal-hallucination claim is not supported by the evidence presented.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents 'Drug Insights,' an open-source retrieval-augmented generation (RAG) chatbot that answers medication queries from a corpus of Nigerian pharmaceutical data, specifically the EMDEX formulary. The system uses Azure OpenAI embeddings, a Pinecone vector database, and GPT-4o for answer generation, with nine prompt variants compared via S-BERT similarity against pharmacist-written answers to 50 queries. The authors report that the best prompt (prompt_0a) achieves an average S-BERT similarity of 84.09%, and they supplement this with pharmacist Likert-scale feedback (averages 3.76–3.88 out of 5). The conclusion claims the system 'delivers accurate, context-specific responses with minimal hallucination.' The paper also describes future plans for retrieval precision/recall evaluation, chat-session testing, and additional similarity metrics.","tokens_in":4924,"tokens_out":2533,"duration_ms":25923,"significance":"The problem addressed is real and consequential: African healthcare workers need rapid, reliable access to medication information, and an open-source, locally relevant RAG tool could help. The authors should be credited for building and releasing the system, grounding it in a real formularies corpus, involving pharmacists in question design and evaluation, and systematically comparing multiple prompt variants. If the accuracy and hallucination claims were rigorously validated, this would be a useful contribution to applied medical NLP and information retrieval. However, the current evaluation does not substantiate those claims: the primary metric (S-BERT similarity to pharmacist answers derived from the same corpus) is not a measure of clinical correctness, the prompt selected as best was chosen on the same test set used to report its score, and the paper itself acknowledges that retrieval recall/precision was not evaluated. The significance is therefore contingent on substantial additional validation.","major_comments":[{"comment":"The central evaluation compares Drug Insights responses with pharmacist answers produced using the same EMDEX corpus and reports S-BERT cosine similarity as the evidence for 'accurate, context-specific responses with minimal hallucination.' This is not a measure of clinical accuracy or hallucination: a semantically similar answer can contain a wrong dosage or contraindication, and a semantically dissimilar answer can be clinically correct. No independent gold standard, sentence-level fact checking, or inter-rater reliability of the pharmacist reference answers is provided. The paper's own Section V lists 'Explicit systematic evaluation of the recall and precision of database retrievals' as future work, which further weakens the current claim.","section":"Section III.B"},{"comment":"The best prompt, prompt_0a, was selected as the one with the highest average S-BERT similarity on the same 50-query set used to report the 84.09% score. This makes the reported number a test-set maximum rather than a held-out estimate, inflating apparent quality. The authors should either split the data into prompt-selection and evaluation sets, use cross-validation, or at minimum report confidence intervals and the selection procedure. Without this, the comparison across prompts in Fig. 2 cannot support any claim about which prompt is best.","section":"Section III.B and Fig. 2"},{"comment":"The retrieval configuration (cosine similarity threshold of 0.9, similarity count of 3) is presented as a fixed design choice without any sensitivity analysis. These are free parameters that can substantially affect retrieval quality, and therefore the final answer. The paper should include a brief sensitivity study or at least a rationale grounded in experiments, especially because Section V acknowledges that retrieval recall/precision has not been evaluated.","section":"Section II.C"},{"comment":"The query set of 50 questions is small, and the paper states that 'a few questions whose answers were not present in the corpus' were included as a hallucination check, but the results for those questions are not reported separately. This is a missed opportunity: reporting how the system handled out-of-corpus queries would speak directly to the hallucination claim. At minimum, the authors should quantify how many such questions were included and show their individual scores or a qualitative assessment.","section":"Section III.B"}],"minor_comments":[{"comment":"The heading 'IV. PREPARE HOW IT WORKS, INCLUDING ARCHITECTURE AND PERFORMANCE METRICS' appears garbled; this likely should be 'SYSTEM OVERVIEW' or 'HOW IT WORKS.'","section":"Section IV heading"},{"comment":"The sentence 'Maintaining the Integrity of the Specifications' appears at the end of Section II.A and is unrelated to the surrounding text; it appears to be leftover boilerplate and should be removed.","section":"Section II.A"},{"comment":"The paper refers to 'Fig 2' when describing the average S-BERT similarity scores, but the figure caption in the text says 'Fig 2' and the later figure is numbered 'Fig 3' while the text refers to 'Fig 3' for the sample query image. The figure numbering and in-text references should be re-checked; also, the sample query image in Fig. 3 is not discussed in the body.","section":"Figures"},{"comment":"The pharmacist feedback table reports averages but not the number of pharmacists who rated the responses, the number of queries each rated, or the standard deviation. Adding these details would make the feedback more interpretable.","section":"Table 2"},{"comment":"The abstract claims the system 'delivers accurate, context-specific responses with minimal hallucination,' but the body results are characterized as 'Preliminary tests' (Abstract) and the conclusion uses 'validated by S-BERT metrics and pharmacist feedback.' The claims should be moderated to match the evidence, e.g., 'shows promise' rather than 'delivers.'","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is an application-oriented systems paper in a workshop-style format, and the engineering effort is real. The key gap is that the primary evaluation does not measure the claimed outcomes (accuracy, minimal hallucination). This is fixable in principle by re-framing the claims and adding a proper evaluation, but the authors will need to do more than minor edits. The journal should weigh whether the current level of validation meets its bar for applied IR contributions; I lean toward giving the authors a chance with a clear revision path."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my honest take on arXiv:2502.15722. It's a straightforward RAG application: extract EMDEX formulary PDFs, chunk them, embed with Azure OpenAI, store in Pinecone, query with GPT-4o, and compare nine zero-shot prompts against pharmacist-written answers using S-BERT similarity. The best prompt scores 84.09%. The paper's real value is in the package: an open-source, working drug-information chatbot for Nigerian healthcare workers, with a corpus that matters locally and a candid discussion of limitations.\n\nWhat's new is not the technique—RAG is standard, and the prompt scan is a parameter sweep. What's useful is that they built it, open-sourced it (github.com/OpenSourceCollective/drug_insights), and tested it in a realistic setting with pharmacist feedback. They also explicitly list in Section V what they did not evaluate: retrieval precision/recall, multi-turn performance, and other similarity metrics. That honesty earns goodwill.\n\nThe soft spot is the evaluation, and it is load-bearing. The abstract claims the system 'delivers accurate, context-specific responses with minimal hallucination.' The only quantitative support is the S-BERT score, which compares the system's answer to a pharmacist's answer, where the pharmacist was restricted to the same EMDEX corpus. That is a consistency check, not a validity check. A semantically similar answer can still contain a wrong dosage or a fabricated contraindication; a valid answer can be phrased very differently. The score says nothing about hallucination. On top of that, prompt_0a was chosen because it performed best on the same 50 queries used to report the 84.09%, so that number is a test-set maximum, not a held-out estimate. There is no error bar, no baseline system, and the human evaluators are affiliated with the funder. These are real issues.\n\nOne small internal contradiction: Section III.B describes prompt_0a as having 'no comparison of multiple generations,' but Section III.A and Table 1 say prompt_0* compares four generated results. Easy to fix, but it suggests the prompt descriptions need a careful pass.\n\nDoes the central argument hold up? As a feasibility study, yes—the system works and the authors were careful to scope their claims in the body. As a demonstration of accuracy and minimal hallucination, no. The evidence does not support that claim. But the authors themselves acknowledge the missing recall/precision work, so they are not hiding it.\n\nWho is this for? Researchers and implementers working on clinical RAG in low-resource settings. It is a decent starting point, not a definitive result.\n\nIf I were editor, I would send it to peer review. The problem is meaningful, the work is reproducible enough (open source), and the evaluation can be strengthened. I would require a major revision: either a proper gold standard (e.g., clinician-verified answers, or at least a statement-level factuality check) or a softened claim that matches the consistency check actually performed.","headline":"A useful open-source RAG pilot for Nigerian formularies, but the evaluation metric under-sells the hallucination claim.","tokens_in":5487,"tokens_out":3367,"would_cite":false,"duration_ms":29342,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An open-source RAG chatbot built on the Nigerian EMDEX formulary returns medication answers that match pharmacist answers at 84% average sentence similarity, supporting faster drug lookups for African healthcare workers.","keywords":["Retrieval Augmented Generation","drug formulary","EMDEX","Nigerian healthcare","prompt engineering","S-BERT evaluation","hallucination reduction","open-source health AI"],"falsifier":"Have an independent clinical panel score each chatbot answer against the full EMDEX formulary and standard references, flagging factual errors, omissions, and unsupported statements. If answers that scored high on semantic similarity to a pharmacist's short answer still contain material factual errors—especially on questions the formulary cannot answer—then the reported 84% similarity score would not establish accurate, low-hallucination drug responses.","tokens_in":4549,"feed_emoji":"💊","tokens_out":10270,"duration_ms":92336,"temperature":0.7,"pith_summary":"Drug Insights is an open-source retrieval-augmented generation (RAG) chatbot that answers medication questions from Nigerian formulary documents. The paper's central claim is that this pipeline—built on the EMDEX essential-medicines formulary, a vector database, and a GPT-based generator—can deliver accurate, context-specific drug information with minimal made-up content, replacing slow manual lookups in printed handbooks. Against 50 doctor- and pharmacist-reviewed queries, the best of nine prompt designs (guardrails, no sentence limit) achieved an average S-BERT similarity score of 84.09% compared with a pharmacist's answer restricted to the same corpus, and the weakest prompt scored 65.91%. Pharmacist ratings of output accuracy, relevance, usefulness, and source relevance averaged about 3.8 out of 5. The paper frames the work as a feasibility demonstration for a prompt-tunable, open-source drug-information tool, with retrieval precision, chat-session behavior, and corpus coverage listed as future work.","feed_headline":"AI chatbot matches pharmacist drug answers at 84%","feed_subtitle":"Drug Insights chatbot built on the Nigerian EMDEX formulary gives African health workers fast, accurate drug answers.","key_machinery":"The load-bearing mechanism is the retrieval-augmented generation loop together with the S-BERT evaluation metric. Sentence-BERT (S-BERT) is a sentence-embedding model that produces a cosine similarity score between two texts, and the paper uses that score to compare Drug Insights' answers with a pharmacist's answers to the same queries. The pipeline itself works through layout-aware text extraction from EMDEX PDFs, schema-based normalization of drug entries by a language model, 1536-dimensional embedding and vector-database storage with top-3 retrieval at a 0.9 cosine threshold, and final answer generation by a GPT-based model from the retrieved passages. Prompt engineering supplies the controlled variation: nine zero-shot prompts differing in output-sentence limits and in whether they use comparison or guardrail instructions, with the guardrail-plus-no-limit combination winning.","core_discovery":"The central claim is that a corpus-bound RAG system can serve as a reliable drug-information lookup for African healthcare workers: given a question, Drug Insights retrieves relevant passages from the EMDEX formulary and generates an answer that, on average, is semantically close to what a pharmacist would write using only that same corpus. The quantitative anchor is the 84.09% average S-BERT cosine similarity achieved by prompt_0a, which pairs guardrail instructions with no sentence limit; the lowest-scoring prompt, prompt_1b, reached 65.91%. Human evaluators from a healthcare-technology partner rated the outputs between 3.76 and 3.88 on a five-point scale for accuracy, relevance, usefulness, and source relevance. The authors interpret these results as evidence that RAG with prompt engineering can reduce hallucination and provide context-limited, trustworthy medication answers.","pith_inferences":["The similarity score measures agreement with a pharmacist who was restricted to the same corpus, so a high score chiefly shows the chatbot can retrieve and restate what a careful reader would find in EMDEX; independent clinical validation would be needed to show the facts themselves are correct.","The paper includes a few questions whose answers are not in the corpus but does not report their scores, so the most direct untested question is whether the system declines to answer or silently fabricates when coverage fails.","The fixed 0.9 cosine threshold and top-3 retrieval are fixed design choices; varying them would separate retrieval quality from generation style as drivers of the 84.09% score.","The evaluation is limited to Nigerian formulary content, so extending the same pipeline to another country's formulary is a natural test of whether the framework generalizes across African healthcare settings."],"forward_implications":["If the 84.09% similarity result holds, the best prompt can produce formulary answers semantically close to a pharmacist's short answer, making the tool usable as a first-pass drug-information resource where pharmacist time is scarce.","Because the framework is open source, other organizations can substitute their own formulary PDFs and reuse the extraction-to-evaluation pipeline without rebuilding it.","The spread from 65.91% to 84.09% across prompt variants implies that prompt design is a primary determinant of RAG answer quality in this setting.","The pharmacist agreement scores around 3.8/5 support using the tool as a first-pass reference, while the paper's stated future work shows it is not yet positioned as a substitute for professional clinical review."],"supporting_citations":[{"why":"Supplies the EMDEX formulary that is the sole knowledge corpus for retrieval and for the pharmacist comparison answers.","marker":"[10]"},{"why":"Defines S-BERT, the sentence-embedding method whose cosine similarity is the paper's main accuracy metric.","marker":"[13]"},{"why":"Provides the prompting techniques used to design the nine prompt variants tested in the evaluation.","marker":"[12]"},{"why":"The prompt-creation tool used to generate the default QA and context prompts for the drug assistant.","marker":"[11]"},{"why":"Affiliated pharmacists produced the comparison answers for S-BERT scoring and the 1–5 usability and accuracy ratings.","marker":"[14]"}],"fun_headline_variants":["RAG chatbot hits 84% pharmacist similarity for drug lookup","Open-source drug bot for Africa scores 84% near pharmacist answers","Drug Insights chatbot: 84% match to pharmacist answers, free","84% close to pharmacists: open-source chatbot for Nigerian meds"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole accuracy claim rests on treating the similarity score between the chatbot's answer and a pharmacist's answer—where the pharmacist could only use the same drug handbook—as proof that the answer is clinically correct and free of made-up information; that equivalence is assumed, not checked against an independent source of truth.","fun_headline_variants_meta":{"raw":{"variants":["RAG chatbot hits 84% pharmacist similarity for drug lookup","Open-source drug bot for Africa scores 84% near pharmacist answers","Drug Insights chatbot: 84% match to pharmacist answers, free","84% close to pharmacists: open-source chatbot for Nigerian meds"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000808,"raw_usage":{"total_tokens":3509,"prompt_tokens":868,"completion_tokens":2641,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":484,"completion_tokens_details":{"reasoning_tokens":2567}},"tokens_in":484,"tokens_out":2641,"duration_ms":18069,"temperature":1.0,"reasoning_tokens":2567,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T11:44:50.156584+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have an independent clinical panel score each chatbot answer against the full EMDEX formulary and standard references, flagging factual errors, omissions, and unsupported statements. If answers that scored high on semantic similarity to a pharmacist's short answer still contain material factual errors—especially on questions the formulary cannot answer—then the reported 84% similarity score would not establish accurate, low-hallucination drug responses.","supporting_citations":[{"cited_title":"EMDEX: The Essential Medicines Index,","cited_arxiv_id":null,"evidence_quote":"Supplies the EMDEX formulary that is the sole knowledge corpus for retrieval and for the pharmacist comparison answers."},{"cited_title":"Sentence -BERT: Sentence Embeddings using Siamese BERT-Networks,","cited_arxiv_id":null,"evidence_quote":"Defines S-BERT, the sentence-embedding method whose cosine similarity is the paper's main accuracy metric."},{"cited_title":"The Prompt Report: A Systematic Survey of Prompting Techniques,","cited_arxiv_id":null,"evidence_quote":"Provides the prompting techniques used to design the nine prompt variants tested in the evaluation."},{"cited_title":"Prompt Artisan,","cited_arxiv_id":null,"evidence_quote":"The prompt-creation tool used to generate the default QA and context prompts for the drug assistant."},{"cited_title":"Advantage Health Africa,","cited_arxiv_id":null,"evidence_quote":"Affiliated pharmacists produced the comparison answers for S-BERT scoring and the 1–5 usability and accuracy ratings."}],"review_version":1}