REVIEW 4 major objections 5 minor 28 references
Arabic Large Language Models for Medical Text Generation
T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read Fine-tuning Mistral-7B-Instruct-v0.2 on a 20,000-pair Arabic medical corpus scraped from social media gives it the best BERTScore among five generative models for Arabic medical text generation.
desk verdict A routine LoRA fine-tuning comparison on a new Arabic medical QA dataset, but the missing train/test split means the headline Mistral advantage may be in-sample memorization, and the clinical-accuracy claims outrun the evidence. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the curated dataset: over 40,000 Arabic social-media medical posts, refined to roughly 20,000 question–answer pairs that retain informal, dialectal language. The method wraps around this corpus in two steps: low-rank adaptation (LoRA) fine-tunes decoder-only generative transformers cheaply, and BERTScore, which compares candidate and reference tokens by semantic embedding similarity, ranks the outputs. The corpus supplies the medical-domain signal, LoRA makes adaptation affordable, and BERTScore turns semantic closeness into a quantitative ranking.
What would settle it
Run the same fine-tuned models on a held-out Arabic medical QA set whose questions were never in the training corpus and whose reference answers were written independently by clinicians; if Mistral-7B no longer clearly beats LLaMA-2-7B, or if clinician review finds high-BERTScore answers are medically wrong, the reported ranking and accuracy claim are refuted.
Extended reading notes
Core claim
The central claim is that fine-tuning Mistral-7B-Instruct-v0.2 on a newly collected Arabic medical corpus makes it the best of five tested generative models for Arabic medical text generation. The paper reports average BERTScore precision of 68.50%, recall of 69.08%, and F1 of 68.50% for Mistral-7B, versus 67.25% F1 for LLaMA-2-7B, 65.07% for GPT-2 Medium, 65.04% for AraGPT2, and 65.60% for BLOOM-560M. The authors also claim the fine-tuned system can provide accurate medical advice, diagnoses, drug recommendations, and treatment plans in response to informal patient input, and that qualitative review supports the coherence and relevance of the generated replies.
Load-bearing premise
The load-bearing premise is that the BERTScore figures measure how well a fine-tuned model answers new patient questions rather than how closely it repeats training examples, and that the social-media doctor replies used as reference answers are medically sound.
Editorial extensions
If this is right
- Mistral-7B-Instruct-v0.2, after LoRA fine-tuning on the Arabic medical corpus, becomes a practical candidate for generating replies in Arabic hospital-management and patient-assistance systems.
- All five models improve their BERTScore after fine-tuning, so the collected corpus has value beyond the winning model, as adaptation data rather than a benchmark.
- The small performance gap among mid-size models means deployment choices can weigh speed and cost: AraGPT2 at 148M parameters stays within about 3.5 F1 points of the 7B model.
- The pipeline of scrape, clean, LoRA fine-tune, and BERTScore evaluation is directly reusable for other dialect-rich, low-resource medical languages.
Reading between the lines
- The paper never states a train/test split for the BERTScore evaluation, so a fair reading treats the numbers as in-sample until a properly held-out split is run; this is the first experiment that would harden or weaken the claim.
- Because the reference answers come from unverified social-media advice, even a high BERTScore cannot establish clinical safety; a clinician-rated gold standard is the missing link between fluent text and accurate medical advice.
- The corpus-construction half of the contribution is probably the most transferable part: the same scrape-and-clean recipe should work for Swahili, Urdu, or other under-resourced medical dialects, possibly with stronger effect than the specific model choice.
- The paper's own stated need for validation mechanisms suggests the 'accurate diagnoses, drugs, and treatment plans' language should be read as a design goal rather than a demonstrated clinical capability.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes an Arabic medical text generation system by fine-tuning five generative LLMs (Mistral-7B-Instruct-v0.2, LLaMA-2-7B, AraGPT2-Base, GPT-2 Medium, and BLOOM-560M) on a newly scraped dataset of roughly 20,000 Arabic question–answer pairs collected from Facebook medical groups. The authors use LoRA for efficient fine-tuning and evaluate the models with BERTScore precision, recall, and F1. They report that Mistral-7B-Instruct-v0.2 achieves the best average F1 of 68.50% and, based on this and qualitative inspection, claim the system can provide accurate medical advice, diagnoses, drug recommendations, and treatment plans.
Significance. If the evaluation were sound, the paper would contribute a new Arabic medical conversation dataset and a comparative benchmark of generative LLMs under BERTScore. The BERTScore equations in Section VI.B are standard and correctly stated, and the choice of models spans useful size and language-coverage ranges. However, the central empirical claim is not supported as reported: the evaluation appears to reuse the fine-tuning dataset without a described held-out split, no clinician or expert validation is provided for the medical-accuracy claims, and the baseline tables are internally inconsistent and lack uncertainty estimates. The dataset itself could be a valuable resource if released with proper documentation, but the current manuscript does not provide enough detail to reproduce or trust the headline results.
major comments (4)
- [§IV, §V.E, §VI.A] The paper never specifies a train/test split. Section IV describes the curated dataset of approximately 20,000 QA pairs used for fine-tuning, and Section VI.A states only that 'Five models were evaluated on subsets of the dataset,' with no mention of a held-out partition, random seed, or selection rule. Since the same dataset is used for fine-tuning, the BERTScore values in Tables I and II may reflect memorization of training data rather than generalization to unseen patient questions. This undermines the central comparative claim that Mistral-7B-Instruct-v0.2 outperforms the other models; the authors must report the evaluation split and re-run the evaluation on a truly held-out subset.
- [Abstract, §III, §VI.D] The abstract claims the system can provide 'accurate medical advice, diagnoses, drug recommendations, and treatment plans,' but no medical expert or clinician validation is reported. The only quantitative evidence is BERTScore, which measures semantic similarity to references scraped from Facebook posts without credential verification; BERTScore is not a measure of medical correctness. Section III itself states that ensuring accuracy requires validation mechanisms such as verified medical sources or expert reviews, yet none are implemented in the paper. The clinical accuracy claims should be removed or supported by appropriate human evaluation.
- [Table I, Table II, §V.C] The baseline and fine-tuned scores are not reproducible or mutually consistent. Table I reports 'Base Model (%)' and 'Fine-Tuned (%)' but does not define which metric (precision, recall, or F1) is shown or how the base models were evaluated (prompt format, decoding strategy, evaluation set). Moreover, for LLaMA-2-7B and BLOOM-560M, the 'Fine-Tuned (%)' values in Table I (66.50 and 64.03) do not match the F1 scores in Table II (67.25 and 65.60). The authors also report single-run numbers with no error bars or significance tests, so the 1.25-point Mistral-versus-LLaMA F1 gap is not established as meaningful. Please clarify the metric and provide uncertainty estimates.
- [§VI.A] The phrase 'subsets of the dataset' is too vague to be reproducible. The number of evaluation examples, the sampling procedure, and whether the subsets overlap the fine-tuning examples are not specified anywhere in the paper. This missing detail is directly tied to the circularity concern in the first major comment and must be documented in full.
minor comments (5)
- [§VI.B, Eqs. (1)-(3)] The symbol S_{i,j} is used in all three BERTScore equations but is never defined. It should be stated as the cosine similarity between contextual token embeddings, and the specific embedding model (e.g., which BERT variant and which language) should be identified for reproducibility.
- [Throughout] The paper alternates between 'BERT Score' and 'BERTScore'. Please use a single spelling consistently, and cite the original BERTScore paper (Zhang et al., 2020) rather than reference [6], which is a medical-chatbot case study and does not introduce the metric.
- [References] Several citation-to-model mappings are incorrect: GPT-2 Medium is cited to [9] (Kühl et al., a general AI/ML paper), and BLOOM-560M is cited to [27] (Nova, a healthcare AI paper). These should point to the actual model resources (e.g., Hugging Face model cards).
- [Table I, §V.C] The caption of Table I should explicitly state the evaluation metric, since Table II reports precision, recall, and F1 separately and the single 'Fine-Tuned (%)' column is ambiguous.
- [Figure 3] Figure 3 is not referenced in the text. Please add a cross-reference and describe what is being compared, since the figure caption alone does not explain the qualitative evaluation.
Circularity Check
Evaluation is run on subsets of the same dataset used for fine-tuning, with no held-out split described; the reported BERTScore gains are in-sample fits rather than out-of-sample predictions.
-
fitted input called prediction
[Section VI.A / Table II; Section IV 'Dataset Collection and Description'; Section V.D 'Training Process']
"Five models were evaluated on subsets of the dataset, primarily due to computational constraints for large models that generate free-form responses [5]."
The fine-tuning step consumes the curated dataset: Section IV says the refined ~20,000 pairs are 'suitable for model training' and Section V.D says 'Each model ... was fine-tuned using the curated Arabic medical dataset.' Section VI.A then evaluates 'subsets of the dataset' with no train/test split, seed, or selection rule described anywhere in Section V.E (only batch size, warmup, FP16, and save strategy are listed). The BERTScore F1 values in Tables I-II are therefore computed against reference answers that are part of (or at least not shown to be outside) the fine-tuning data.
full rationale
The paper's central benchmark claim is that fine-tuned Mistral-7B-Instruct-v0.2 outperforms the other models on Arabic medical text generation, with average BERTScore F1 of 68.50%. However, the evaluation protocol is circular in the classic machine-learning sense: the models are fine-tuned on the curated dataset described in Section IV, and Section VI.A states that all five models were evaluated on 'subsets of the dataset.' No held-out partition, train/test split, random seed, or selection rule is described; Section V.E lists only training hyperparameters and save/evaluation strategy. Consequently, Tables I and II compare the models' outputs against reference answers drawn from the same distribution (at minimum, not shown to be excluded) as the fine-tuning data, so the reported F1 values and the Mistral advantage measure in-sample fit or memorization rather than generalization to unseen patient questions. The paper's clinical accuracy claims (diagnoses, drug recommendations, treatment plans) are also unsupported by the BERTScore evidence, but that is a validity/correctness problem rather than a separate circularity. The author self-citations in the reference list are peripheral and not load-bearing for the central result. The score of 6 reflects one significant circular step that undermines the main comparative claim, while the derivation of BERTScore equations themselves is standard and non-circular.
Assumptions & free parameters
free parameters (3)
- Evaluation subset selection =
not specified
- BERTScore embedding model =
not specified
- Training hyperparameters (LoRA rank, learning rate, epochs) =
not specified
assumptions (5)
- domain assumption Facebook posts from medical groups contain sufficiently accurate and safe medical advice to serve as training and evaluation ground truth.
- domain assumption The preprocessing pipeline retains a representative 20,000 QA pairs without introducing bias.
- domain assumption BERTScore semantic similarity is a valid proxy for medical response quality.
- domain assumption Low-rank adaptation preserves the base model's capabilities for this task.
- standard math BERTScore equations (1)-(3) are the standard definitions of the metric.
Cite this review
Pith. "Pith review of Arabic Large Language Models for Medical Text Generation." pith.science (2026). https://pith.science/paper/3F577RDT
@misc{pith2026250910095,
author = {Pith},
title = {Pith review of: Arabic Large Language Models for Medical Text Generation},
year = {2026},
howpublished = {\url{https://pith.science/paper/3F577RDT}},
note = {Machine review of arXiv:2509.10095}
}
read the original abstract
Efficient hospital management systems (HMS) are critical worldwide to address challenges such as overcrowding, limited resources, and poor availability of urgent health care. Existing methods often lack the ability to provide accurate, real-time medical advice, particularly for irregular inputs and underrepresented languages. To overcome these limitations, this study proposes an approach that fine-tunes large language models (LLMs) for Arabic medical text generation. The system is designed to assist patients by providing accurate medical advice, diagnoses, drug recommendations, and treatment plans based on user input. The research methodology required the collection of a unique dataset from social media platforms, capturing real-world medical conversations between patients and doctors. The dataset, which includes patient complaints together with medical advice, was properly cleaned and preprocessed to account for multiple Arabic dialects. Fine-tuning state-of-the-art generative models, such as Mistral-7B-Instruct-v0.2, LLaMA-2-7B, and GPT-2 Medium, optimized the system's ability to generate reliable medical text. Results from evaluations indicate that the fine-tuned Mistral-7B model outperformed the other models, achieving average BERT (Bidirectional Encoder Representations from Transformers) Score values in precision, recall, and F1-scores of 68.5\%, 69.08\%, and 68.5\%, respectively. Comparative benchmarking and qualitative assessments validate the system's ability to produce coherent and relevant medical replies to informal input. This study highlights the potential of generative artificial intelligence (AI) in advancing HMS, offering a scalable and adaptable solution for global healthcare challenges, especially in linguistically and culturally diverse environments.
Figures
Reference graph
Works this paper leans on
-
[1]
The computer-based patient record: an essential technology for health care,
D. E. Detmer, E. B. Steen, and R. S. Dick, “The computer-based patient record: an essential technology for health care,” 1997
work page 1997
-
[2]
The concept of value in resource allocation,
T. C. Brown, “The concept of value in resource allocation,”Land economics, vol. 60, no. 3, pp. 231–246, 1984
work page 1984
-
[3]
M. Rashid and M. Sharma, “Ai-assisted diagnosis and treatment plan- ning—a discussion of how ai can assist healthcare professionals in making more accurate diagnoses and treatment plans for diseases,”AI in Disease Detection: Advancements and Applications, pp. 313–336, 2025
work page 2025
-
[4]
Large language models in healthcare and medical domain: A review,
Z. A. Nazi and W. Peng, “Large language models in healthcare and medical domain: A review,” inInformatics, vol. 11, no. 3. MDPI, 2024, p. 57
2024
-
[5]
A study of generative large language model for medical research and healthcare,
C. Peng, X. Yang, A. Chen, K. E. Smith, N. PourNejatian, A. B. Costa, C. Martin, M. G. Flores, Y . Zhang, T. Magocet al., “A study of generative large language model for medical research and healthcare,” NPJ digital medicine, vol. 6, no. 1, p. 210, 2023
work page 2023
-
[6]
Enhancing healthcare management: A case study of a medical chatbot in egypt,
L. A. Salem, T. Shishtawy, N. El-Attaret al., “Enhancing healthcare management: A case study of a medical chatbot in egypt,”Benha Journal of Applied Sciences, vol. 9, no. 5, pp. 199–210, 2024
work page 2024
-
[7]
Llm-sem: A sentiment- based student engagement metric using llms for e-learning platforms,
A. Hamdi, A. A. Mazrou, and M. Shaltout, “Llm-sem: A sentiment- based student engagement metric using llms for e-learning platforms,” arXiv preprint arXiv:2412.13765, 2024
arXiv 2024
-
[8]
A. Hamdi, H. Kassab, M. Bahaa, and M. Mohamed, “Riro: Reshaping inputs, refining outputs unlocking the potential of large language models in data-scarce contexts,”arXiv preprint arXiv:2412.15254, 2024
arXiv 2024
Show all 28 references
-
[9]
Artificial intel- ligence and machine learning,
N. K ¨uhl, M. Schemmer, M. Goutier, and G. Satzger, “Artificial intel- ligence and machine learning,”Electronic Markets, vol. 32, no. 4, pp. 2235–2244, 2022
2022
-
[10]
Hospital management system with chatbot,
M. L. PRIY ANKA, B. T. R. NAY AKA, B. S. RAKSHITHA, B. V AMSHI, and B. M. K. Y ADA V , “Hospital management system with chatbot,”International Journal of Mechanical Engineering Research and Technology, vol. 16, no. 2, pp. 144–154, 2024
2024
-
[11]
Deep learning for arabic healthcare: Medicalbot,
M. Abdelhay, A. Mohammed, and H. A. Hefny, “Deep learning for arabic healthcare: Medicalbot,”Social Network Analysis and Mining, vol. 13, no. 1, p. 71, 2023
2023
-
[12]
Nabiha: an arabic dialect chat- bot,
D. Al-Ghadhban and N. Al-Twairesh, “Nabiha: an arabic dialect chat- bot,”International Journal of Advanced Computer Science and Appli- cations, vol. 11, no. 3, 2020
2020
-
[13]
Lmrpa: Large language model-driven efficient robotic process automation for ocr,
O. Hosam Abdellaif, A. Nader, and A. Hamdi, “Lmrpa: Large language model-driven efficient robotic process automation for ocr,”arXiv e- prints, pp. arXiv–2412, 2024
2024
-
[14]
Enhancing hospital bed management through chatbots: Inte- grating dialogflow and firebase cloud messaging for real-time bed avail- ability and reservation system,
K. Namee, T. Prempranee, W. Phonngam, R. Kaewsaeng-On, and A. Meny, “Enhancing hospital bed management through chatbots: Inte- grating dialogflow and firebase cloud messaging for real-time bed avail- ability and reservation system,” in2024 22nd International Conference on ICT ...
2024
-
[15]
How valuable is medical social media data? content analysis of the medical web,
K. Denecke and W. Nejdl, “How valuable is medical social media data? content analysis of the medical web,”Information Sciences, vol. 179, no. 12, pp. 1870–1880, 2009
2009
-
[16]
Health topics on facebook groups: content analysis of posts in multiple sclerosis communities,
S. Della Rosa, F. Senet al., “Health topics on facebook groups: content analysis of posts in multiple sclerosis communities,”Interactive Journal of Medical Research, vol. 8, no. 1, p. e10146, 2019
2019
-
[17]
Erpa: Efficient rpa model integrating ocr and llms for intelligent document processing,
O. H. Abdellaif, A. N. Hassan, and A. Hamdi, “Erpa: Efficient rpa model integrating ocr and llms for intelligent document processing,” in2024 International Mobile, Intelligent, and Ubiquitous Computing Conference (MIUCC). IEEE, 2024, pp. 295–300
2024
-
[18]
Generative ai in medicine and healthcare: promises, opportunities and challenges,
P. Zhang and M. N. Kamel Boulos, “Generative ai in medicine and healthcare: promises, opportunities and challenges,”Future Internet, vol. 15, no. 9, p. 286, 2023
2023
-
[19]
Lmrpa: Large lan- guage model-driven efficient robotic process automation for ocr,
O. H. Abdellaif, A. Nader, and A. Hamdi, “Lmrpa: Large lan- guage model-driven efficient robotic process automation for ocr,”arXiv preprint arXiv:2412.18063, 2024
2024 arXiv
-
[20]
Lmv-rpa: Large model voting- based robotic process automation,
O. Abdellatif, A. Ayman, and A. Hamdi, “Lmv-rpa: Large model voting- based robotic process automation,”arXiv preprint arXiv:2412.17965, 2024
2024 arXiv
-
[21]
A medical chatbot for tunisian dialect using a rule-based and machine learning approach,
A. Rekik, M. Chebbi, and N. Boughanmi, “A medical chatbot for tunisian dialect using a rule-based and machine learning approach,” Journal of Medical Informatics, vol. 12, no. 3, pp. 45–56, 2023
2023
-
[22]
Llm-daas: Llm-driven drone-as-a-service operations from text user requests,
L. Wassim, K. Mohamed, and A. Hamdi, “Llm-daas: Llm-driven drone-as-a-service operations from text user requests,”arXiv preprint arXiv:2412.11672, 2024
2024 arXiv
-
[23]
Pharmago: An online phar- maceutical ordering platform,
R. G. Gamage, K. U. Senadeera, N. S. Bandara, K. Y . Abeywardena, D. D. Diyamullage, and N. Amarasena, “Pharmago: An online phar- maceutical ordering platform,” in2021 3rd International Conference on Advancements in Computing (ICAC), 2021, pp. 364–370
2021
-
[24]
Towards building a chatbot-based first aid service in arabic language,
J. Anwar, P. Nadi, and N. Seddik, “Towards building a chatbot-based first aid service in arabic language,”Journal of Advanced Research in Applied Sciences and Engineering Technology, vol. 45, no. 2, pp. 1–10, 2025
2025
-
[25]
Asem: Enhancing empathy in chatbot through attention-based sentiment and emotion modeling,
O. Hamad, K. Shaban, and A. Hamdi, “Asem: Enhancing empathy in chatbot through attention-based sentiment and emotion modeling,” in Proceedings of the 2024 Joint International Conference on Computa- tional Linguistics, Language Resources and Evaluation (LREC-COLING 2024), 2024,...
2024
-
[26]
Using ai to automatically process data from unstructured health records of patients with lung cancer,
M. Aldea, P. Rolland, S. Simon, A. Poplu, M. Wartelle, B. Vignal, J.-C. Louis, F. Lion, A. Borie, D. Planchardet al., “Using ai to automatically process data from unstructured health records of patients with lung cancer,”Cancer Research, vol. 84, no. 6 Supplement, pp. 3569–3569, 2024
2024
-
[27]
Generative ai in healthcare: advancements in electronic health records, facilitating medical languages, and personalized patient care,
K. Nova, “Generative ai in healthcare: advancements in electronic health records, facilitating medical languages, and personalized patient care,” Journal of Advanced Analytics in Healthcare Management, vol. 7, no. 1, pp. 115–131, 2023
2023
-
[28]
Medimatch: Ai-driven drug recommendation system,
S. Khan, A. Saify, S. Gosaliya, D. Jain, and M. J. Zalte, “Medimatch: Ai-driven drug recommendation system,” in2024 2nd International Conference on Sustainable Computing and Smart Systems (ICSCSS). IEEE, 2024, pp. 1342–1349
2024
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.