Lightweight Clinical Decision Support System using QLoRA-Fine-Tuned LLMs and Retrieval-Augmented Generation

Aditya Varma; Anil S. Mokhade; Mohammad Shoaib Ansari; Mohd Sohail Ali Khan; Shubham Revankar

REVIEW 3 major objections 5 minor 12 references

Lightweight Clinical Decision Support System using QLoRA-Fine-Tuned LLMs and Retrieval-Augmented Generation

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that a compact 3-billion-parameter Llama model, fine-tuned with QLoRA and fed by retrieval over hospital-specific documents, can make clinical decision support practical in low-resource settings, with benchmark evidence…

desk verdict A reproducible QLoRA fine-tuning benchmark for medical QA wrapped in an untested RAG framing; the benchmark is fine, the headline claim isn't supported. read the letter →

arxiv 2505.03406 v1 pith:IA5G6POK submitted 2025-05-06 cs.CL cs.AI

Mohammad Shoaib Ansari , Mohd Sohail Ali Khan , Shubham Revankar , Aditya Varma , Anil S. Mokhade This is my paper · ORCID

classification cs.CLcs.AI

keywords clinicaldecisionsupportretrieval-augmentedgenerationQLoRALlama3.2parameter-efficientfine-tuningmedicalquestionansweringlow-resourcedeploymentMedMCQA

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper is trying to establish that a small, efficient language model can serve as a practical clinical decision-support system when two ingredients are combined: QLoRA fine-tuning, which adapts a 3-billion-parameter Llama 3.2 model to medical text while keeping memory and weights small, and retrieval-augmented generation (RAG), which pulls relevant hospital-specific guidelines, protocols, and records into each prompt. The authors argue that this combination gives hospitals with limited computing budgets a way to get institution-aware answers without retraining or deploying a much larger model. Their quantitative evidence is a benchmark comparison in which the fine-tuned model beats the untuned base model on MedMCQA (56.39% versus 50.9%) and on most MMLU medical subsets, though it trails on a few. The paper also illustrates the intended behavior with sample workflows, such as a pediatric diabetic ketoacidosis query answered from an institution-specific protocol, and stresses that outputs are decision-support suggestions requiring clinician review.

What carries the argument

Two mechanisms carry the argument. QLoRA is parameter-efficient fine-tuning that keeps the base model's weights frozen in 4-bit quantized form and trains only small low-rank adapter matrices, replacing the layer update $y = Wx + b$ with $y = (W + BA)x + b$; the paper reports that this cuts memory needs from roughly 2 GB to 0.5 GB per 1 GB of model and, in this implementation, leaves only 2.4 million trainable parameters with peak training memory near 4.3 GB. RAG is the retrieval pipeline: hospital documents are segmented into roughly 512-token chunks with metadata, embedded with E5-large-v2, indexed in a vector database, and retrieved at query time by cosine similarity combined with BM25 lexical search and optional metadata and recency filters; the retrieved segments then enter a prompt template that instructs the model to consider hospital protocols first and to distinguish retrieved context from general knowledge. The benchmark numbers in Table 3 compare the QLoRA-tuned model directly with the untuned Llama 3.2-3B-Instruct base, which is the only part of the accuracy claim that is experimentally measured.

What would settle it

Run the paper's pipeline on a fixed set of clinical queries under three conditions—fine-tuned model alone, fine-tuned model with random retrieved context, and fine-tuned model with top-k retrieved context—and have clinicians score answer accuracy and protocol adherence; if the retrieved-context condition does not beat the model-alone condition, or if random context performs equally well, the central RAG claim fails.

Watch

Extended reading notes

Core claim

The paper's central claim is that integrating quantized low-rank fine-tuning with retrieval-augmented generation produces a lightweight clinical decision-support system that improves the accuracy and local relevance of medical answers. Fine-tuning Llama 3.2-3B-Instruct on 26,412 question–answer pairs from the Medical Meadow WikiDoc and MedQuAD datasets, using rank-8 LoRA adapters with alpha 16 and about 0.75% of parameters trainable, lifts MedMCQA accuracy from 50.9% to 56.39% and improves most MMLU medical subsets relative to the base model. The RAG component is the second half of the claim: hospital documents are chunked, embedded, and indexed so that runtime queries retrieve the top 5–10 relevant segments, which are assembled into a prompt instructing the model to prioritize hospital protocols and attribute sources. The paper's demonstration of this component is illustrative rather than measured: a sample query shows a generic non-RAG response replaced by one citing a hospital's June 2023 pediatric DKA protocol, with institution-specific dosing, monitoring intervals, and ICU transfer criteria.

Load-bearing premise

The system's clinical-value claim depends on the unmeasured assumption that retrieved hospital-specific documents materially improve the model's answers; if retrieval is noisy or never changes the output, the RAG half of the claimed advantage collapses.

Editorial extensions

If this is right

A hospital with a single 24 GB consumer-grade GPU could reproduce the fine-tuning step, since reported peak reserved memory was about 4.3 GB and total training time was under two hours.
If the RAG design works as intended, updating the hospital vector database with new guidelines would update the system's answers without retraining the model.
The benchmark gains imply that a 3-billion-parameter model adapted this way can outperform its untuned base on medical multiple-choice tasks, so smaller models are a viable starting point for institution-specific deployment.
In the authors' intended workflow, every output remains a clinician-reviewed suggestion, with the prompt explicitly asking the model to flag information gaps and cite sources.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper's abstract claim that RAG 'significantly improves response accuracy' is not directly tested; an A/B comparison of the same fine-tuned model with and without retrieved context, scored by clinicians, would be the natural experiment to support or refute that claim.
If retrieval noise is low, the same architecture could be translated across institutions by re-embedding a new hospital corpus, turning 'hospital-specific' decision support into a configuration change rather than a retraining effort.
A useful testable extension is to measure how often retrieved context actually changes the model's output, since the illustrative DKA example cannot establish whether hospital protocols alter answers in routine use.
The benchmark results support basic medical question-answering capability, not clinical safety; a prospective pilot in which clinicians rate differential diagnoses and summaries against expert standards would be the bridge to practice.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Read this as a system-building report, not a validation paper. The QLoRA fine-tuning half is a clear, reproducible benchmark study; the RAG half of the title is essentially unevaluated. If the authors had only claimed "we fine-tuned a 3B model with QLoRA and measured it on MedMCQA and MMLU medical subsets," the paper would largely deliver. The claim that retrieval of hospital-specific context improves answers is not tested anywhere.

What's good: the architecture is concrete—E5-large-v2 embeddings, chunking, vector DB, hybrid retrieval, prompt structure—and the training details are unusually complete. The paper reports rank, alpha, learning rate, epochs, loss curves, runtime, GPU memory, and throughput. The benchmark evaluation is independent of the training data (WikiDoc + MedQuAD in; MedMCQA + MMLU out), so no circularity. The fine-tuned 3B model beats the base on most subsets, with the biggest gains on College Biology (78.74 vs 70.83) and MedMCQA (56.39 vs 50.9). That is a legitimate, if incremental, empirical contribution.

The soft spots are real and central. The abstract and conclusion claim RAG "significantly improves response accuracy," but Table 3 compares fine-tuned vs base model without retrieval. The DKA example in Section 4.3 is illustrative, not measured; there are no retrieval precision/recall numbers, no RAG-vs-non-RAG comparison, and no end-to-end evaluation on hospital data. "Significant" is also used without significance testing, and the fine-tuned model is worse than base on MMLU College Medicine and Medical Genetics. The paper overstates "superior performance compared to existing models"—no external baselines are given. Code is not released, which limits reproducibility despite the detailed hyperparameters.

These are the usual gaps for a systems paper, but they matter more here because the RAG claim is the advertised value proposition. The fine-tuning part could stand alone as a modest benchmark report. I'd send this to a serious referee mainly because the empirical core is honest and the low-resource clinical deployment angle is worth engaging. The referee should ask for a RAG ablation, retrieval quality metrics, error bars, and code release before acceptance. Who benefits: readers working on lightweight medical LLM deployment who want a concrete QLoRA recipe; readers looking for validated clinical decision support will come away unsatisfied.

Referee Report

3 major / 5 minor

Summary. The paper proposes a clinical decision support system that combines QLoRA fine-tuning of Llama 3.2-3B-Instruct with retrieval-augmented generation over hospital-specific data. The authors describe the architecture for document preprocessing, embedding, and hybrid retrieval, and report fine-tuning on a combined WikiDoc and MedQuAD dataset of 26,412 question-answer pairs. Evaluation is performed on MedMCQA and selected MMLU medical subsets, with the fine-tuned model compared against the base model in Table 3. The paper also contains qualitative use-case descriptions for disease prediction, treatment suggestion, and medical report summarization, along with discussion of ethical and deployment considerations.

Significance. If the claimed benefits were empirically validated, a lightweight QLoRA-fine-tuned LLM combined with institution-specific RAG could be genuinely useful in low-resource clinical settings, and the paper's emphasis on memory efficiency and deployment is relevant. However, as written, the manuscript's quantitative contribution is limited to a single fine-tuning experiment with mixed results and no statistical support. The central RAG claim is not tested at all. The paper is better characterized as a system description with preliminary fine-tuning results than as a validated decision-support system.

major comments (3)

[Abstract; §4.3] The abstract and conclusion claim that retrieval-augmented generation "significantly improves response accuracy," but no experiment in the paper compares RAG-enabled outputs with non-RAG outputs. Table 3 evaluates only the QLoRA-fine-tuned model versus the base Llama 3.2-3B-Instruct on MedMCQA and MMLU subsets, and the RAG prompt example in §4.3 is an illustrative narrative, not a measured output of the implemented system. The RAG pipeline in §2 is described but never evaluated with retrieval quality metrics or an end-to-end comparison. This leaves the paper's central advertised benefit without empirical support. I recommend adding an ablation that measures response accuracy with and without retrieved context on a held-out clinical QA set, or substantially revising the abstract and conclusion to restrict claims to the QLoRA fine-tuning component.
[Table 3; §3.2.4] The claim of "significant improvements" is not supported by the reported numbers. The fine-tuned model is worse than the base model on MMLU College Medicine (56.07 vs. 58.38) and Medical Genetics (71.00 vs. 74.00), and the Professional Medicine improvement (74.63 vs. 74.26) is within likely run-to-run variability. No standard deviations, confidence intervals, number of test examples, or significance tests are reported, and only one run appears to have been performed. Please report evaluation details, including exact prompts, accuracy computation, and sample counts, and provide uncertainty measures or avoid the word "significant" unless a statistical test supports it.
[§7 Conclusion] The conclusion asserts "superior performance of our model compared to existing models," but the only empirical comparison in the paper is against the base Llama model in Table 3. No comparisons with prior medical QA systems, alternative fine-tuning methods, or RAG baselines are provided. Please restrict the conclusion to claims that are directly supported by the presented experiments, or add the missing baselines.

minor comments (5)

[§3.2.1] Please clarify the composition of the 26,412-question training set: how many examples come from each source, and was any deduplication or overlap check against the MedMCQA and MMLU evaluation sets performed?
[§2.1] The retrieval top-k is described as "typically k=5-10, adjusted based on query complexity," but no sensitivity analysis or explanation of how k was chosen in any experiment is given.
[§3.2.3] Table 2 lists a "Learning Rate" of 1.2195e-7 while the text states an initial learning rate of 2e-4 with a linear schedule; please clarify whether this is the final learning rate or a typo.
[Figure 2] Please ensure the loss and learning-rate curves have clearly labeled axes and legends; the current figure is hard to interpret in the text alone.
[§5.1.3] The example workflow gives probability estimates (78%, 42%, 18%) without indicating how these were computed; please state whether these are illustrative or measured outputs and, if measured, provide the underlying evidence.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: benchmark evaluations are independent measurements, and the unsupported RAG claim is a missing experiment, not a circular derivation.

full rationale

The paper's quantitative claims rest on Table 3, where the QLoRA-fine-tuned Llama 3.2-3B-Instruct is compared with the base model on MedMCQA and MMLU medical subsets. These are external benchmarks, and the fine-tuning data (WikiDoc and MedQuAD, Section 3.2.1) are disjoint from the evaluation sets, so the reported accuracy improvements do not reduce to the training inputs by construction. The RAG component is described architecturally (Sections 2.2 and 4) and illustrated with a hypothetical DKA example (Section 4.3), but no experiment compares RAG versus non-RAG outputs and no retrieval metrics are reported. That is an evidentiary gap: the abstract's claim that the system 'significantly improves response accuracy' via RAG is unsupported, not circular. There are no fitted parameters renamed as predictions, no uniqueness theorems imported from the authors' prior work, and no self-citation chains carrying a load-bearing conclusion. The cited works are external datasets and standard method papers (QLoRA, LoRA, MedMCQA, MMLU), which provide independent grounding. Accordingly, the paper receives a circularity score of 0, with the caveat that the central RAG benefit remains unvalidated rather than derivationally circular.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claim depends on several unverified domain assumptions about benchmark transfer, embedding quality, and quantization fidelity, plus hand-chosen hyperparameters. These are not derived or validated within the paper.

free parameters (5)

LoRA rank (r) = 8
Chosen by hand in Section 3.2.2; controls adapter capacity and affects fine-tuning results.
LoRA alpha = 16
Scaling factor for LoRA updates, set in Section 3.2.2 without reported tuning.
Learning rate = 2e-4
Initial learning rate for AdamW (Section 3.2.3); selected manually, affects final loss and benchmark outcomes.
Training epochs = 1
Single epoch over 26,412 pairs (Section 3.2.3); no epoch tuning or early stopping reported.
Retrieval top-k = 5-10 (variable)
System parameter for RAG (Section 2.2); its value is not optimized or evaluated.

assumptions (4)

domain assumption Benchmark accuracy on MedMCQA and MMLU medical subsets is a valid proxy for clinical decision support quality
The paper's clinical utility claim rests on these benchmarks; no clinician validation is performed.
domain assumption The E5-large-v2 embedding model captures clinically meaningful semantics for hospital document retrieval
Invoked in Sections 2.1 and 4.1 without evaluation on the hospital corpus.
domain assumption Fine-tuning on Medical Meadow WikiDoc and MedQuAD generalizes to diverse clinical queries
The training distribution is assumed sufficient; no out-of-distribution evaluation is provided.
domain assumption 4-bit NF4 quantization preserves the medical information needed for accurate responses
Asserted in Section 3.3 but not directly compared with unquantized fine-tuning.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Lightweight Clinical Decision Support System using QLoRA-Fine-Tuned LLMs and Retrieval-Augmented Generation." pith.science (2026). https://pith.science/paper/IA5G6POK

@misc{pith2026250503406,
  author       = {Pith},
  title        = {Pith review of: Lightweight Clinical Decision Support System using QLoRA-Fine-Tuned LLMs and Retrieval-Augmented Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IA5G6POK}},
  note         = {Machine review of arXiv:2505.03406}
}

read the original abstract

This research paper investigates the application of Large Language Models (LLMs) in healthcare, specifically focusing on enhancing medical decision support through Retrieval-Augmented Generation (RAG) integrated with hospital-specific data and fine-tuning using Quantized Low-Rank Adaptation (QLoRA). The system utilizes Llama 3.2-3B-Instruct as its foundation model. By embedding and retrieving context-relevant healthcare information, the system significantly improves response accuracy. QLoRA facilitates notable parameter efficiency and memory optimization, preserving the integrity of medical information through specialized quantization techniques. Our research also shows that our model performs relatively well on various medical benchmarks, indicating that it can be used to make basic medical suggestions. This paper details the system's technical components, including its architecture, quantization methods, and key healthcare applications such as enhanced disease prediction from patient symptoms and medical history, treatment suggestions, and efficient summarization of complex medical reports. We touch on the ethical considerations-patient privacy, data security, and the need for rigorous clinical validation-as well as the practical challenges of integrating such systems into real-world healthcare workflows. Furthermore, the lightweight quantized weights ensure scalability and ease of deployment even in low-resource hospital environments. Finally, the paper concludes with an analysis of the broader impact of LLMs on healthcare and outlines future directions for LLMs in medical settings.

Figures

Figures reproduced from arXiv: 2505.03406 by the authors.

**Figure 1.** System Architecture Diagram 2.1 Hospital Data Preprocessing and Embedding The system’s ability to provide contextually relevant responses is rooted in its integration with hospitalspecific data, which include clinical guidelines, electronic health records (EHRs), treatment protocols, and institutional best practices. Crucially, when incorporating sensitive sources like EHRs, appropriate data de-identification techn… view at source ↗

**Figure 2.** Loss Value and Learning Rate during Finetuning [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

**Figure 3.** Data Preprocessing Pipeline for RAG 4.2 Hybrid Retrieval System Our RAG implementation employs a hybrid retrieval approach that combines multiple retrieval mechanisms to optimize for different types of medical queries. This includes vector similarity search as the 6 [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

12 extracted references · 3 canonical work pages

[1]

Retrieval augmented generation for 10 large language models and its generalizability in assessing medical fitness,

Y. H. Ke, L. Jin, K. Elangovan, et al. , “Retrieval augmented generation for 10 large language models and its generalizability in assessing medical fitness,” en, npj Digital Medicine, vol. 8, no. 1, p. 187, Apr. 2025, issn: 2398-6352. doi: 10.1038/s41746-025-01519-z

doi:10.1038/s41746-025-01519-z 2025
[2]

Lewis, E

P. Lewis, E. Perez, A. Piktus, et al. , Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, arXiv:2005.11401, Apr. 2021. doi: 10.48550/arXiv.2005.11401. 11

arXiv doi:10.48550/arxiv.2005.11401 2005
[3]

L. Wang, N. Yang, X. Huang, et al. , Text Embeddings by Weakly-Supervised Contrastive Pre- training, arXiv:2212.03533 version: 2, Feb. 2024. doi: 10.48550/arXiv.2212.03533

arXiv doi:10.48550/arxiv.2212.03533 2024
[4]

Dettmers, A

T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer, QLoRA: Efficient Finetuning of Quan- tized LLMs, arXiv:2305.14314, May 2023. doi: 10.48550/arXiv.2305.14314

arXiv doi:10.48550/arxiv.2305.14314 2023
[5]

E. J. Hu, Y. Shen, P. Wallis, et al. , LoRA: Low-Rank Adaptation of Large Language Models , arXiv:2106.09685 version: 2, Oct. 2021. doi: 10.48550/arXiv.2106.09685

arXiv doi:10.48550/arxiv.2106.09685 2021
[6]

Medalpaca/medicalmeadow wikidoc · Datasets at Hugging Face , Apr. 2025. [Online]. Available: https://huggingface.co/datasets/medalpaca/medical_meadow_wikidoc

work page 2025
[7]

[Online]

Healthcare NLP: LLMs, Transformers, Datasets , en. [Online]. Available: https://www.kaggle. com/datasets/jpmiller/layoutlm

work page
[8]

A. Pal, L. K. Umapathi, and M. Sankarasubbu, MedMCQA : A Large-scale Multi-Subject Multi- Choice Dataset for Medical domain Question Answering , arXiv:2203.14371, Mar. 2022. doi: 10. 48550/arXiv.2203.14371

arXiv doi:10.48550/arxiv.2203.14371 2022

Show all 12 references

[9]

Hendrycks, C

D. Hendrycks, C. Burns, S. Basart, et al., Measuring Massive Multitask Language Understanding , arXiv:2009.03300 version: 3, Jan. 2021. doi: 10.48550/arXiv.2009.03300

2009 doi arXiv
[10]

Current and future state of evaluation of large language models for medical summarization tasks,

E. Croxford, Y. Gao, N. Pellegrino, et al., “Current and future state of evaluation of large language models for medical summarization tasks,” en, npj Health Systems , vol. 2, no. 1, p. 6, Feb. 2025, issn: 3005-1959. doi: 10.1038/s44401-024-00011-2

2025 doi
[11]

A Review of Large Language Models in Medical Education, Clinical Decision Support, and Healthcare Administration,

J. Vrdoljak, Z. Boban, M. Vilovi´ c, M. Kumri´ c, and J. Boˇ zi´ c, “A Review of Large Language Models in Medical Education, Clinical Decision Support, and Healthcare Administration,” en, Healthcare, vol. 13, no. 6, p. 603, Mar. 2025, issn: 2227-9032. doi: 10.3390/healthcare13060603

2025 doi
[12]

Ethical Considerations in the Use of Artificial Intelligence and Machine Learning in Health Care: A Comprehensive Review,

M. Harishbhai Tilala, P. Kumar Chenchala, A. Choppadandi, et al., “Ethical Considerations in the Use of Artificial Intelligence and Machine Learning in Health Care: A Comprehensive Review,” en, Cureus, Jun. 2024, issn: 2168-8184. doi: 10.7759/cureus.62443. 12

2024 doi

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

Retrieval augmented generation for 10 large language models and its generalizability in assessing medical fitness,

Y. H. Ke, L. Jin, K. Elangovan, et al. , “Retrieval augmented generation for 10 large language models and its generalizability in assessing medical fitness,” en, npj Digital Medicine, vol. 8, no. 1, p. 187, Apr. 2025, issn: 2398-6352. doi: 10.1038/s41746-025-01519-z

doi:10.1038/s41746-025-01519-z 2025

[2] [2]

Lewis, E

P. Lewis, E. Perez, A. Piktus, et al. , Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, arXiv:2005.11401, Apr. 2021. doi: 10.48550/arXiv.2005.11401. 11

arXiv doi:10.48550/arxiv.2005.11401 2005

[3] [3]

L. Wang, N. Yang, X. Huang, et al. , Text Embeddings by Weakly-Supervised Contrastive Pre- training, arXiv:2212.03533 version: 2, Feb. 2024. doi: 10.48550/arXiv.2212.03533

arXiv doi:10.48550/arxiv.2212.03533 2024

[4] [4]

Dettmers, A

T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer, QLoRA: Efficient Finetuning of Quan- tized LLMs, arXiv:2305.14314, May 2023. doi: 10.48550/arXiv.2305.14314

arXiv doi:10.48550/arxiv.2305.14314 2023

[5] [5]

E. J. Hu, Y. Shen, P. Wallis, et al. , LoRA: Low-Rank Adaptation of Large Language Models , arXiv:2106.09685 version: 2, Oct. 2021. doi: 10.48550/arXiv.2106.09685

arXiv doi:10.48550/arxiv.2106.09685 2021

[6] [6]

Medalpaca/medicalmeadow wikidoc · Datasets at Hugging Face , Apr. 2025. [Online]. Available: https://huggingface.co/datasets/medalpaca/medical_meadow_wikidoc

work page 2025

[7] [7]

[Online]

Healthcare NLP: LLMs, Transformers, Datasets , en. [Online]. Available: https://www.kaggle. com/datasets/jpmiller/layoutlm

work page

[8] [8]

A. Pal, L. K. Umapathi, and M. Sankarasubbu, MedMCQA : A Large-scale Multi-Subject Multi- Choice Dataset for Medical domain Question Answering , arXiv:2203.14371, Mar. 2022. doi: 10. 48550/arXiv.2203.14371

arXiv doi:10.48550/arxiv.2203.14371 2022