Preserving Privacy, Increasing Accessibility, and Reducing Cost: An On-Device Artificial Intelligence Model for Medical Transcription and Note Generation

Ayush Mudgal; Dhruv Diddi; Johnson Thomas; Nisten Tahiraj; Wendao Liu; Zeeshaan Mohammed

REVIEW 3 major objections 5 minor 24 references

Preserving Privacy, Increasing Accessibility, and Reducing Cost: An On-Device Artificial Intelligence Model for Medical Transcription and Note Generation

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Fine-tuning a one-billion-parameter LLM with LoRA on 1,500 synthetic transcripts yields better, safer structured medical notes that still run entirely in the browser.

desk verdict A useful artifact release wrapped in a self-referential evaluation; the clinically meaningful claim is not supported, but the model and data are worth a second look. read the letter →

arxiv 2507.03033 v1 pith:KF3ELOIO submitted 2025-07-03 cs.CL cs.AI

Johnson Thomas , Ayush Mudgal , Wendao Liu , Nisten Tahiraj , Zeeshaan Mohammed , Dhruv Diddi This is my paper

classification cs.CLcs.AI

keywords medicaltranscriptionSOAPnoteson-deviceAIprivacy-preservinghealthcareparameter-efficientfine-tuningLoRAsyntheticclinicaldatahallucinations

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that a deliberately small language model can do useful medical transcription work entirely on a user's own device. The authors take a one-billion-parameter base model and adapt it with LoRA on 1,500 synthetic endocrinology consultations, producing a system that turns transcripts into structured notes. Against the untuned model, the adapted model scores higher on ROUGE and BERTScore, is judged more factually correct and complete, and produces far fewer major hallucinations and omissions. The point of the size constraint is deployment: at one billion parameters the model can run in a browser, so patient data never leaves the clinic and there are no per-use cloud costs. If these results hold beyond the synthetic evaluation sets, they point to a practical route for private, low-cost clinical documentation.

What carries the argument

The load-bearing object is the LoRA-adapted one-billion-parameter model: LoRA, or low-rank adaptation, adds a small set of trainable low-rank matrices to frozen weights, so the whole adapted model stays near one billion parameters and can be loaded in a browser. The other machinery is the evaluation stack: ROUGE, BERTScore, and BLEURT measure surface and semantic similarity to reference notes; an LLM-as-judge rates factual correctness, completeness, relevance, coherence, terminology, and readability; and a separate categorization labels each output's hallucinations and omissions as none, minor, or major. The safety categorization is what turns a text-quality comparison into a claim about clinical risk.

What would settle it

A clinician-blinded study on real outpatient endocrinology visits would settle the claim: collect unmodified transcripts, have the base and fine-tuned models generate notes, and have independent clinicians score correctness, completeness, and hallucinations; if the fine-tuned model's advantage disappears or reverses, the central claim is falsified. A cheaper proxy would be to swap the LLM judge for a different judge model and compare scores on the same outputs.

Watch

Extended reading notes

Core claim

The central claim is that parameter-efficient fine-tuning of a compact LLM for medical transcription yields clinically meaningful improvements while enabling complete on-device browser deployment. Concretely, the authors report that their LoRA-tuned one-billion-parameter model outperforms the base model on both a 100-transcript internal endocrinology set and a 140-case modified external benchmark. On the external set, ROUGE-1 rises from 0.346 to 0.496, BERTScore F1 from 0.832 to 0.866, LLM-judged factual correctness from 2.81 to 3.54, major hallucinations fall from 85 to 35 cases, and major omissions from 107 to 21. On the internal set the composite clinical-quality score rises from 3.13 to 4.43. The authors interpret these gains as evidence that domain-specific adaptation, not model scale, is the main driver, and that the same adapted model remains small enough to run locally in the browser.

Load-bearing premise

The results stand or fall on whether the synthetic transcripts and modified reference notes used for training and testing really behave like real doctor-patient conversations, and whether the automated judge's idea of a good note matches a clinician's.

Editorial extensions

If this is right

If the central claim holds, medical transcription can run with complete data sovereignty: transcription and note generation on local hardware, with no patient data sent to a vendor.
A one-billion-parameter model removes the per-token or per-seat cloud charges that price smaller practices out of AI scribes, making documentation assistance more like buying software than buying compute.
The reported reduction in major hallucinations and omissions, if reliable, addresses a known failure mode of automated notes; the paper cites evidence that human notes themselves contain at least one error and four omissions on average.
The recipe of synthetic domain pairs plus LoRA appears to transfer from the training specialty to an external benchmark, suggesting the same adaptation step could be repeated for other specialties.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A direct next test would pair this transcript-to-note model with an on-device speech recognizer, creating a fully local pipeline from raw audio to finished note; the paper stops at transcripts.
The same synthetic-data-and-LoRA recipe could be reapplied to other specialties and documentation formats such as history-and-physical notes or discharge summaries, where a similar gain pattern would confirm that the mechanism is domain adaptation.
Because the adapted model is small enough for a browser, it could be embedded directly into electronic health record interfaces, making AI documentation a built-in feature rather than a separate cloud service.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Jane: You should know two things about arXiv:2507.03033. First, the artifacts are real and useful: a LoRA-tuned Llama 3.2 1B for endocrinology note generation, a 1,500-pair synthetic training set, evaluation data, and a browser demo. Second, the central conclusion — that fine-tuning yields clinically meaningful improvements — is not actually supported by the experiments as run.

What the paper does well: it is a straightforward application of PEFT to a compact model, and the authors release the model, data, and evaluation code. That is exactly the kind of reproducibility affordance we want. The direction of the metric changes is consistent: ROUGE, BERTScore, and LLM-judged quality all go up relative to the base model, and the hallucination/omission counts drop. The on-device browser deployment angle is a nice practical touch for small practices.

The soft spots are substantial and they sit right on the load-bearing claim. Training data (§3.2) are synthetic transcripts generated by an LLM workflow. The internal evaluation set (§3.3, item 1) is, by the paper's own words, 'specifically created for this project' — presumably the same workflow. The ACI benchmark references were 'modified to match our training data format.' So the large gains in Table 1 may reflect format imitation and train/eval distribution overlap, not better clinical documentation. All quality, hallucination, and omission scores come from gpt-4.1-mini; no clinician read the model outputs. The 20-sample endocrinologist check validated the data-generation methodology, not the outputs on real encounters.

The paper's own limitations section concedes that the LLM-as-judge may not capture all aspects of clinical utility apparent to human clinicians, and that evaluation on curated datasets may not reflect real-world transcription. Those admissions are honest but they undercut the abstract's 'clinically meaningful' phrasing. There are also no confidence intervals or significance tests, and BLEURT is flat or slightly worse on the ACI set — minor, but worth noting.

Bottom line: this is not a desk reject. It is a competent, transparent empirical application with real artifacts and a testable claim. What it needs is a human-clinician evaluation on unmodified real transcripts, or at minimum a much clearer argument that the synthetic distribution is not doing the work, plus proper statistical support. I'd send it to peer review with the expectation of major revision. The model and dataset deserve citing even if the paper's headline claim needs more evidence.

Referee Report

3 major / 5 minor

Summary. This manuscript describes an on-device medical transcription system built by LoRA fine-tuning Llama 3.2 1B on 1,500 synthetic endocrinology transcripts. The authors evaluate the fine-tuned OnDevice model against the base model on 100 internally generated synthetic transcripts and 140 ACI benchmark transcripts whose reference notes were reformatted to match the training format. They report large improvements in ROUGE/BERTScore, GPT-4.1-mini-judged factual correctness and completeness, and large reductions in judged major hallucinations and omissions, and conclude that fine-tuning compact LLMs yields clinically meaningful improvements while enabling complete on-device browser deployment.

Significance. The engineering contribution is real: the authors demonstrate that a 1B model with LoRA can be adapted to a structured-note task and released openly, including the model, datasets, evaluation code, and judge prompts. The direction of the results is plausible and the on-device/privacy framing is relevant. However, the significance for clinical practice is currently not established, because every effectiveness measure is either computed on the training distribution or assigned by an unvalidated LLM judge. The paper would be genuinely significant if the same gains were shown on unmodified real transcripts with clinician adjudication.

major comments (3)

[§3.2 and §3.3, Tables 1–3] The training data (§3.2), the internal evaluation set (§3.3, item 1), and the ACI benchmark references (item 2, modified to match the training data format) all come from the same synthetic or format-aligned pipeline. The large ROUGE/BERTScore gains in Table 1 (e.g., internal ROUGE-1 0.363→0.653) may therefore reflect that the fine-tuned model has learned the exact reference template and topic distribution, rather than improved clinical documentation. A specific test: evaluate the fine-tuned model on an unmodified, independently sourced set of real, or at least not template-aligned, transcripts and report ROUGE/BERTScore. Without such a test, the central claim of "clinically meaningful improvements" is not supported.
[§3.4 and Table 3] All clinical quality, hallucination, and omission scores are produced by gpt-4.1-mini. The 20-sample endocrinologist check in §3.2 validated the data-generation methodology, not the model's outputs on either evaluation set. Since the judge may prefer outputs that match the training format, the large hallucination and omission reductions in Table 3 (e.g., 85→35 major hallucinations on ACI) are not independently established. The paper's own limitations section concedes that the LLM-as-judge evaluation may not capture clinical utility apparent to human clinicians. Clinician or at least independently adjudicated evaluation of model outputs is required to support the abstract's and conclusion's wording.
[§5 and §6] The conclusion states "consistent improvements across automated metrics" and "clinically meaningful improvements," but the manuscript does not report confidence intervals, significance tests, or inter-rater agreement for the LLM judge, and BLEURT slightly decreases on the ACI benchmark (Table 1). The discussion's "robust assessment methodology" claim is therefore overstated; the results are internally consistent and suggestive, but the aggregate scores are not characterized statistically. This matters because the headline improvements are used to justify the conclusion.

minor comments (5)

[§3.4] The sentence "GPT 4.1 mini evaluated the LLM generated notes for following using a Likert scale like rating" should be rewritten; please specify the exact judge prompt, the rubric for each dimension, and whether each output was scored with a single pass or multiple samples.
[§4.1] "Table 1 and figure 1 presents" should be "present"; figure captions and in-text references for Figures 1–3 are missing, making it difficult to connect the figures to the reported values.
[§5] The claim that human-generated clinical notes have "at least 1 error and 4 omissions" should be checked against reference [24]; the phrase "on average" as currently written may overstate the source's finding.
[Data Availability] The dataset URLs contain the Unicode ligature character "ﬁ" in "starﬁshdata"; those URLs will break or be copy-pasted incorrectly and should use plain ASCII "fi".
[§3.2] The statement that 1,500 samples are "sufficient for parameter-efficient fine-tuning" would be easier to assess if the LoRA hyperparameters (rank, alpha, learning rate), batch size, and number of epochs were reported.

Circularity Check

2 steps flagged · score 6.0 of 10

Training and evaluation draw from the same synthetic generation workflow, and the external benchmark's references were rewritten to the model's training format, so key reported gains are substantially self-referential rather than independent evidence of clinical improvement.

fitted input called prediction [Section 3.2 (Dataset Preparation) and Section 3.3 (Evaluation Framework), item 1]
"Training data consisted of 1,500 synthetic medical transcriptions and corresponding structured medical notes specifically created for this project and available publicly [17]. ... Internal Evaluation Dataset: 100 synthetic transcripts and structured medical notes specifically created for this project and publicly available [18]."

The same synthetic endocrinology generation-and-refinement workflow used for the 1,500 training pairs is used to create the 100 internal evaluation transcripts. The fine-tuned model is therefore evaluated on the same distribution and note format it was optimized to reproduce. Its large gains on this set (ROUGE-1 0.363->0.653; composite 3.13->4.43) are measurements of distribution/format imitation rather than independent evidence of clinically meaningful transcription quality. The 20-sample endocrinologist check validated the data-generation methodology, not the model's outputs, so it does not break the circle.
fitted input called prediction [Section 3.3 (Evaluation Framework), item 2]
"Modified ACI Benchmark: 140 transcripts from ACI bench, with structured notes modified to match our training data format, also publicly available [19]."

The ACI reference notes are rewritten to match the exact structured format the model was LoRA-tuned to emit. Consequently, the fine-tuned model's higher ROUGE-1 (0.346->0.496), ROUGE-2 (0.118->0.227), and BERTScore (0.832->0.866) on the ACI dataset partially reward the model for conforming to its own training format. This removes format independence from the external benchmark: the comparison is partially circular. The ACI transcripts themselves are external, which limits the circularity, but the reference modification makes the metric gains partly a measure of training-format adherence rather than clinical content.

full rationale

The central claim of 'clinically meaningful improvements' rests on evaluations whose reference standards are coupled to the training pipeline. The internal evaluation set is generated by the same project as the training data, and the ACI references are explicitly modified to match the training data format. Thus the large ROUGE/BERTScore gains and even some LLM-judge quality gains reflect in part the model reproducing the synthetic distribution and format it was fitted to produce. The base-model comparison and the external ACI transcripts provide some independent grounding, and the paper is transparent about its synthetic data and LLM-as-judge limitations. However, the paper's own limitation statement concedes that 'the LLM-as-judge evaluation, while comprehensive, may not capture all aspects of clinical utility that would be apparent to human clinicians' and that 'the evaluation was conducted on curated datasets and may not fully reflect the challenges of real-world clinical transcription environments.' These concessions, combined with the same-distribution internal eval and format-modified external references, make the headline conclusion substantially self-referential rather than a fully independent demonstration. No evidence of classic self-citation circularity or imported uniqueness theorems was found.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The central claim rests on the fidelity of synthetic data, the clinical validity of an LLM judge, and the unmodified correctness of reformatted ACI references; all three are assumed rather than independently demonstrated. The hand-chosen LoRA configuration is the main unreported free parameter.

free parameters (1)

LoRA hyperparameters (rank, alpha, learning rate) = not reported in manuscript
Chosen by hand to make the fine-tuning work; values are not stated, and output quality depends on them.

assumptions (3)

domain assumption LLM-generated synthetic transcripts and notes are faithful to real clinical conversations and documentation.
Section 3.2; only 20 of 1,500 samples were reviewed by an endocrinologist, and there is no comparison to real transcription data.
domain assumption GPT-4.1-mini ratings of factual correctness, completeness, hallucinations, and omissions are valid proxies for clinical assessment.
Section 3.4; the judge is an LLM, and its outputs were not validated against human clinicians on the final evaluation sets.
domain assumption Reformatting ACI reference notes to match the training format does not change their correctness as ground truth.
Section 3.3; the modification procedure is not described and the modified references were not independently checked.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Preserving Privacy, Increasing Accessibility, and Reducing Cost: An On-Device Artificial Intelligence Model for Medical Transcription and Note Generation." pith.science (2026). https://pith.science/paper/KF3ELOIO

@misc{pith2026250703033,
  author       = {Pith},
  title        = {Pith review of: Preserving Privacy, Increasing Accessibility, and Reducing Cost: An On-Device Artificial Intelligence Model for Medical Transcription and Note Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KF3ELOIO}},
  note         = {Machine review of arXiv:2507.03033}
}

read the original abstract

Background: Clinical documentation represents a significant burden for healthcare providers, with physicians spending up to 2 hours daily on administrative tasks. Recent advances in large language models (LLMs) offer promising solutions, but privacy concerns and computational requirements limit their adoption in healthcare settings. Objective: To develop and evaluate a privacy-preserving, on-device medical transcription system using a fine-tuned Llama 3.2 1B model capable of generating structured medical notes from medical transcriptions while maintaining complete data sovereignty entirely in the browser. Methods: We fine-tuned a Llama 3.2 1B model using Parameter-Efficient Fine-Tuning (PEFT) with LoRA on 1,500 synthetic medical transcription-to-structured note pairs. The model was evaluated against the base Llama 3.2 1B on two datasets: 100 endocrinology transcripts and 140 modified ACI benchmark cases. Evaluation employed both statistical metrics (ROUGE, BERTScore, BLEURT) and LLM-as-judge assessments across multiple clinical quality dimensions. Results: The fine-tuned OnDevice model demonstrated substantial improvements over the base model. On the ACI benchmark, ROUGE-1 scores increased from 0.346 to 0.496, while BERTScore F1 improved from 0.832 to 0.866. Clinical quality assessments showed marked reduction in major hallucinations (from 85 to 35 cases) and enhanced factual correctness (2.81 to 3.54 on 5-point scale). Similar improvements were observed on the internal evaluation dataset, with composite scores increasing from 3.13 to 4.43 (+41.5%). Conclusions: Fine-tuning compact LLMs for medical transcription yields clinically meaningful improvements while enabling complete on-device browser deployment. This approach addresses key barriers to AI adoption in healthcare: privacy preservation, cost reduction, and accessibility for resource-constrained environments.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

24 extracted references · 14 canonical work pages

[1]

Introduction The administrative burden of clinical documentation has reached crisis proportions in modern healthcare, with physicians dedicating 50% of their time to electronic health record (EHR) tasks[1]. This documentation burden contributes significantly to physician burnout, with studies indicating that for every hour of direct patient care, clinicia...

work page
[2]

Google's Med-PaLM and Med-PaLM 2 demonstrated that LLMs could achieve physician-level performance on medical question-answering tasks [8]

Related Work 2.1 Medical Transcription with Large Language Models The application of LLMs to healthcare has gained significant traction in recent years. Google's Med-PaLM and Med-PaLM 2 demonstrated that LLMs could achieve physician-level performance on medical question-answering tasks [8]. Similarly, Microsoft's BioGPT showed promising results in biomedi...

work page
[3]

Fine-tuning was performed using PEFT with LoRA adaptation, implemented through the Unsloth library for accelerated training and merging

Methods 3.1 Model Architecture and Fine-tuning We employed Llama 3.2 1B Instruct as our base model, chosen for its balance of capability and computational efficiency suitable for on-device deployment. Fine-tuning was performed using PEFT with LoRA adaptation, implemented through the Unsloth library for accelerated training and merging. 3.2 Dataset Prepara...

work page
[4]

Internal Evaluation Dataset: 100 synthetic transcripts and structured medical notes specifically created for this project and publicly available [18]

work page
[5]

Modified ACI Benchmark: 140 transcripts from ACI bench, with structured notes modified to match our training data format, also publicly available [19]. 3.4 Evaluation Metrics Text Similarity Metrics: ● ROUGE: We calculated ROUGE-1, ROUGE-2, ROUGE-L, and ROUGE-Lsum to measure n-gram overlap between generated and reference notes [20]. ● BERTScore: This metr...

work page 2025
[6]

Results 4.1 Text Similarity Metrics Performance Table 1 and figure 1 presents the text similarity evaluation results comparing the base Llama 3.2 1B model with our fine-tuned OnDevice model across both evaluation datasets. Table 1: Automated Evaluation Results Dataset Model ROUGE-1 ROUGE-2 ROUGE-L BERTScore F1 BLEURT ACI Benchmark Base_Llama 0.346 0.118 0...

work page
[7]

Discussion Our evaluation demonstrates that the OnDevice model achieved consistent performance improvements across datasets, with particularly notable enhancements in clinical safety through reduced hallucinations and omissions. Given that studies estimate human-generated clinical notes have, on average, at least 1 error and 4 omissions, our focus on redu...

work page
[8]

Conclusion This study demonstrates that fine-tuning compact large language models for structured medical note creation can achieve substantial improvements in clinical documentation quality while enabling complete on-device deployment. The OnDevice model showed consistent improvements across automated metrics, clinical quality assessments, and safety meas...

2017

Show all 24 references

[9]

Allocation of Physician Time in Ambulatory Practice: A Time and Motion Study in 4 Specialties

Sinsky C, Colligan L, Li L, Prgomet M, Reynolds S, Goeders L, Westbrook J, Tutty M, Blike G. Allocation of Physician Time in Ambulatory Practice: A Time and Motion Study in 4 Specialties. Ann Intern Med. 2016 Dec 6;165(11):753-760. doi: 10.7326/M16-0961. Epub 2016 Sep 6. PMID:...

2016 doi
[10]

Impact of Electronic Health Record Use on Cognitive Load and Burnout Among Clinicians: Narrative Review

Asgari E, Kaur J, Nuredini G, Balloch J, Taylor AM, Sebire N, Robinson R, Peters C, Sridharan S, Pimenta D. Impact of Electronic Health Record Use on Cognitive Load and Burnout Among Clinicians: Narrative Review. JMIR Med Inform. 2024 Apr 12;12:e55499. doi: 10.2196/55499. PMID...

2024 doi
[11]

Roles and Potential of Large Language Models in Healthcare: A Comprehensive Review

Lin C, Kuo CF. Roles and Potential of Large Language Models in Healthcare: A Comprehensive Review. Biomed J. 2025 Apr 29:100868. doi: 10.1016/j.bj.2025.100868. Epub ahead of print. PMID: 40311872

2025
[12]

Privacy preserving strategies for electronic health records in the era of large language models

Jonnagaddala, J., Wong, Z.SY. Privacy preserving strategies for electronic health records in the era of large language models. npj Digit. Med. 8, 34 (2025). https://doi.org/10.1038/s41746-025-01429-0 6. https://ai.meta.com/blog/llama-3-2-connect-2024-vision-edge-mobile-devices...

2025 arXiv
[13]

Toward expert-level medical question answering with large language models

Singhal K, Tu T, Gottweis J, Sayres R, Wulczyn E, Amin M, Hou L, Clark K, Pfohl SR, Cole-Lewis H, Neal D, Rashid QM, Schaekermann M, Wang A, Dash D, Chen JH, Shah NH, Lachgar S, Mansﬁeld PA, Prakash S, Green B, Dominowska E, Agüera Y Arcas B, Tomašev N, Liu Y, Wong R, Semturs ...

2025
[14]

MedAlpaca -- An Open-Source Collection of Medical Conversational AI Models and Training Data

Renqian Luo, Liai Sun, Yingce Xia, Tao Qin, Sheng Zhang, Hoifung Poon, Tie-Yan Liu, BioGPT: generative pre-trained transformer for biomedical text generation and mining, Brieﬁngs in Bioinformatics, Volume 23, Issue 6, November 2022, bbac409, https://doi.org/10.1093/bib/bbac409...

2022 arXiv
[15]

The Sound of Healthcare: Improving Medical Transcription ASR Accuracy with Large Language Models

Adedeji, Ayo et al. “The Sound of Healthcare: Improving Medical Transcription ASR Accuracy with Large Language Models.” ArXiv abs/2402.07658 (2024): n. Pag

2024 arXiv
[16]

SpecialtyScribe: Enhancing SOAP note Scribing for Medical Specialties using LLMs

Rastogi, Eti, et al. "SpecialtyScribe: Enhancing SOAP note Scribing for Medical Specialties using LLMs." Proceedings of the Eighteenth ACM International Conference on Web Search and Data Mining. 2025

2025
[17]

Federated machine learning in healthcare: A systematic review on clinical applications and technical architecture

Teo ZL, Jin L, Li S, Miao D, Zhang X, Ng WY, Tan TF, Lee DM, Chua KJ, Heng J, Liu Y, Goh RSM, Ting DSW. Federated machine learning in healthcare: A systematic review on clinical applications and technical architecture. Cell Rep Med. 2024 Feb 20;5(2):101419. doi: 10.1016/j.xcrm...

2024
[18]

KidneyTalk-open: No-code Deployment of a Private Large Language Model with Medical Documentation-Enhanced Knowledge Database for Kidney Disease

Long, Yongchao, et al. "KidneyTalk-open: No-code Deployment of a Private Large Language Model with Medical Documentation-Enhanced Knowledge Database for Kidney Disease." arXiv preprint arXiv:2503.04153 (2025)

2025
[19]

Medicine on the Edge: Comparative Performance Analysis of On-Device LLMs for Clinical Reasoning, author={Leon Nissen and Philipp Zagar and Vishnu Ravi and Aydin Zahedivash and Lara Marie Reimer and Stephan Jonas and Oliver Aalami and Paul Schmiedmayer}, year={2025}, eprint={25...

2025 arXiv
[20]

Rouge: A package for automatic evaluation of summaries

https://huggingface.co/datasets/starﬁshdata/endocrinology_structured_notes_1500/tree/main/data 18. https://huggingface.co/datasets/starﬁshdata/endocrinology_structured_notes_eval 19. https://huggingface.co/datasets/starﬁshdata/endocrinology_structured_notes_aci_bench_eval 20. ...

2004
[21]

Bertscore: Evaluating text generation with bert

Zhang, Tianyi, et al. "Bertscore: Evaluating text generation with bert." arXiv preprint arXiv:1904.09675 (2019)

2019 arXiv
[22]

BLEURT: Learning robust metrics for text generation

Sellam, Thibault, Dipanjan Das, and Ankur P. Parikh. "BLEURT: Learning robust metrics for text generation." arXiv preprint arXiv:2004.04696 (2020)

2020 arXiv
[23]

Analyzing and interpreting data from likert-type scales

Sullivan GM, Artino AR Jr. Analyzing and interpreting data from likert-type scales. J Grad Med Educ. 2013 Dec;5(4):541-2. doi: 10.4300/JGME-5-4-18. PMID: 24454995; PMCID: PMC3886444

2013 doi
[24]

Human evaluation and correlation with automatic metrics in consultation note generation

Moramarco, Francesco, et al. "Human evaluation and correlation with automatic metrics in consultation note generation." arXiv preprint arXiv:2204.00447 (2022)

2022 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

Introduction The administrative burden of clinical documentation has reached crisis proportions in modern healthcare, with physicians dedicating 50% of their time to electronic health record (EHR) tasks[1]. This documentation burden contributes significantly to physician burnout, with studies indicating that for every hour of direct patient care, clinicia...

work page

[2] [2]

Google's Med-PaLM and Med-PaLM 2 demonstrated that LLMs could achieve physician-level performance on medical question-answering tasks [8]

Related Work 2.1 Medical Transcription with Large Language Models The application of LLMs to healthcare has gained significant traction in recent years. Google's Med-PaLM and Med-PaLM 2 demonstrated that LLMs could achieve physician-level performance on medical question-answering tasks [8]. Similarly, Microsoft's BioGPT showed promising results in biomedi...

work page

[3] [3]

Fine-tuning was performed using PEFT with LoRA adaptation, implemented through the Unsloth library for accelerated training and merging

Methods 3.1 Model Architecture and Fine-tuning We employed Llama 3.2 1B Instruct as our base model, chosen for its balance of capability and computational efficiency suitable for on-device deployment. Fine-tuning was performed using PEFT with LoRA adaptation, implemented through the Unsloth library for accelerated training and merging. 3.2 Dataset Prepara...

work page

[4] [4]

Internal Evaluation Dataset: 100 synthetic transcripts and structured medical notes specifically created for this project and publicly available [18]

work page

[5] [5]

Modified ACI Benchmark: 140 transcripts from ACI bench, with structured notes modified to match our training data format, also publicly available [19]. 3.4 Evaluation Metrics Text Similarity Metrics: ● ROUGE: We calculated ROUGE-1, ROUGE-2, ROUGE-L, and ROUGE-Lsum to measure n-gram overlap between generated and reference notes [20]. ● BERTScore: This metr...

work page 2025

[6] [6]

Results 4.1 Text Similarity Metrics Performance Table 1 and figure 1 presents the text similarity evaluation results comparing the base Llama 3.2 1B model with our fine-tuned OnDevice model across both evaluation datasets. Table 1: Automated Evaluation Results Dataset Model ROUGE-1 ROUGE-2 ROUGE-L BERTScore F1 BLEURT ACI Benchmark Base_Llama 0.346 0.118 0...

work page

[7] [7]

Discussion Our evaluation demonstrates that the OnDevice model achieved consistent performance improvements across datasets, with particularly notable enhancements in clinical safety through reduced hallucinations and omissions. Given that studies estimate human-generated clinical notes have, on average, at least 1 error and 4 omissions, our focus on redu...

work page

[8] [8]

Conclusion This study demonstrates that fine-tuning compact large language models for structured medical note creation can achieve substantial improvements in clinical documentation quality while enabling complete on-device deployment. The OnDevice model showed consistent improvements across automated metrics, clinical quality assessments, and safety meas...

2017