Pith. sign in

REVIEW 3 major objections 5 minor 35 references

Are LLMs reliable? An exploration of the reliability of large language models in clinical note generation

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that all 12 tested LLMs are semantically stable in clinical note generation, that open-weight Llama 3.1 70B and Mistral Small are the most reliable models, and that these can be locally deployed while preserving patient…

desk verdict Useful benchmark of 12 LLMs for clinical note generation, but the 'most reliable' ranking is not supported by the numbers as reported. read the letter →

arxiv 2505.17095 v1 pith:GQW7BTG2 submitted 2025-05-21 cs.CL

Kristine Ann M. Carandang , Jasper Meynard P. Araña , Ethan Robert A. Casin , Christopher P. Monterola , Daniel Stanley Y. Tan , Jesus Felix B. Valenzuela , Christian M. Alis This is my paper
classification cs.CL
keywords clinicalnotegenerationlargelanguagemodelreliabilityintra-promptstabilitysemanticconsistencyBERTScoreopen-weightLLMsaci-benchhealthcaredataprivacy
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether large language models can be trusted to write clinical notes from doctor-patient conversation transcripts. Twelve models from Anthropic, Meta, Mistral, and OpenAI were each run ten times on the same 112-transcript prompt, and the repeated outputs were scored for string identity, semantic consistency, and similarity to expert-written notes. The headline finding is that all models are semantically stable, with median semantic consistency above 96% for every model, so the same clinical meaning recurs even when wording differs. Meta's Llama 3.1 70B is rated the most reliable overall, followed by Mistral's Small model, and both open-weight models outperform the proprietary ones. The paper concludes that these smaller open-weight models can be deployed locally for clinical note generation, which keeps protected health information out of third-party APIs.

What carries the argument

The load-bearing mechanism is a three-metric reliability protocol using ten repeated generations from the same prompt. Consistency rate (CR) counts how often two generated notes are string-identical, semantic consistency (SC) measures the average cosine similarity, via BERTScore, between all pairs of generated notes, and semantic similarity (SS) measures BERTScore between a generated note and the expert ground-truth note. BERTScore, which prior clinical note work found suitable for this task, is the device that turns 'same meaning' and 'correctness' into numbers, and the deterministic model settings (temperature 0, top_p 0, top_k 1 where available) are what make the repeated runs comparable as a test of stability rather than sampling noise.

What would settle it

Re-score a random sample of generated notes on the aci-bench subset using a clinical-domain embedding model and have two or more clinicians independently mark critical errors; if the clinician rankings of model quality differ from the BERTScore ranking, or if notes with BERTScore above 80 contain critical omissions like a missed medication, then the claim that semantic consistency and semantic similarity certify reliability for clinical note generation would be refuted.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that intra-prompt reliability for clinical note generation holds at the semantic level. Exact string output varied hugely between models, from 0% to 100% median consistency rate, yet every model's median semantic consistency exceeded 96%, and five models reached 100% semantic consistency. Correctness, measured as median BERTScore similarity between generated notes and expert ground-truth notes, ranged from about 80 to 88 across models, which the paper notes is higher than a previously reported best BERTScore of 60.8 on the same dataset. Meta's Llama 3.1 70B combines perfect semantic consistency with the best joint position on stability and correctness, Mistral's Small model is second, and both sit above all proprietary models. The paper presents this as evidence that reliable, privacy-preserving clinical note generation is achievable with locally deployed open-weight models.

Load-bearing premise

The study's load-bearing premise is that BERTScore, computed with an unspecified HuggingFace checkpoint, correctly captures both the sameness of meaning across repeated notes and the clinical accuracy of a note relative to an expert note; if that embedding-based score misses clinically decisive errors such as a missing medication or a wrong dosage, the reported reliability ratings and the model ranking may not reflect clinical safety.

Editorial extensions

If this is right

  • If the reliability ranking is correct, healthcare providers can run Llama 3.1 70B or Mistral Small locally for note generation and avoid sending protected health information to proprietary API endpoints.
  • Because all models stayed above 96% semantic consistency, string equivalence is not a necessary condition for reliable note generation; semantic consistency alone can serve as the monitoring metric for intra-prompt stability.
  • The BERTScore values in the low-to-mid 80s across all models, well above the 60.8 earlier baseline, indicate that current LLM-generated notes are meaningfully closer to expert notes than earlier in-context-learning results on the same dataset.
  • Within the Anthropic, Meta, and Mistral families, smaller models matched or beat their larger siblings on stability, while correctness favored larger models, implying that model choice for clinical note generation involves a separate scale-stability tradeoff for each family.
  • The recommendation to deploy open-weight models locally directly supports compliance with data privacy regulations in healthcare settings, since submitted prompts to proprietary platforms may be retained for training.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If BERTScore is accepted as the right proxy, the paper's results imply that the remaining barrier to clinical use of LLM note generation is correctness rather than stability, pointing future work toward clinical validation, human expert review, and error analysis instead of consistency tuning.
  • Because the BERTScore checkpoint is unspecified, the model ranking reported here may be checkpoint-dependent; recomputing the twelve models with a clinical-domain embedding model would test whether Llama 70B's lead over GPT-4o persists or narrows.
  • The paper's local-deployment recommendation presumes that the privacy and cost benefits of open weights outweigh the operational burden of self-hosting; for low-resource settings, the edge-size Llama 1B and 3B models that achieved perfect semantic consistency but lower semantic similarity could offer a practical tradeoff that the paper does not quantify.
  • Generalizing beyond the aci-bench role-play transcripts, the near-universal high semantic consistency suggests that other long-form clinical summarization tasks, such as discharge summaries or consultation notes in other languages, would also show high stability, though correctness would need to be evaluated separately.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper reports an empirical study of 12 large language models (Anthropic, Meta, Mistral, OpenAI) for clinical note generation (CNG) from the aci-bench dataset. Using 112 doctor-patient transcripts and 10 iterations of the same prompt per transcript, it measures three reliability-related quantities: consistency rate (exact string equality), semantic consistency (BERTScore among a model's repeated outputs), and semantic similarity (BERTScore against expert ground-truth notes). The main reported findings are that all models achieve median semantic consistency above 96%, that three models achieve perfect consistency rate, and that Meta's Llama 3.1 70B and Mistral Small are the most reliable models overall, leading the authors to recommend local deployment of relatively small open-weight models for CNG to address data-privacy concerns.

Significance. The paper addresses a genuine gap: empirical evidence on the intra-prompt reliability of open-weight LLMs for clinical note generation. Its methodological strengths include the use of an external public benchmark with expert ground-truth notes, 112 transcripts, 10 repeated runs per transcript, documented deterministic settings for the models, and clear reporting of medians and interquartile ranges for each metric. If the central findings held, they would be practically useful for privacy-conscious local deployment of open-weight models. However, the headline ranking and the 'correctness' interpretation currently rest on statistical comparisons and BERTScore calibration that are not yet established, and there are internal inconsistencies in the reported numbers. These issues need to be addressed before the practical recommendation can be accepted.

major comments (3)
  1. [Section 4.3 and Table 2] The claim that Llama 3.1-70B is 'most reliable considering both semantic consistency and semantic similarity' lacks a defined aggregation rule and statistical support. In Table 2, Llama-70B has SC=100 and SS=85.90±1.29, while GPT-4o-mini has SC=97.40 and SS=87.26±1.24 and ChatGPT-4o has SC=97.52 and SS=87.01±1.33. With no weighting specified and with overlapping IQRs, the asserted ranking could be reversed under an equally plausible weighting of stability versus correctness. I recommend adding an explicit composite reliability index or a paired statistical comparison over the 112 per-transcript values (for example, Wilcoxon signed-rank tests on SC and SS separately, or a paired bootstrap of the rank ordering) before drawing the 'most reliable' conclusion in Sections 4.3 and 5.
  2. [Table 2 versus Figure 3] The numeric results are internally inconsistent between Table 2 and Figure 3. For example, the median semantic similarity for Llama 3.1-70B is reported as 85.90±1.29 in Table 2 but 87.56±1.4 in Figure 3; Claude Haiku is 85.61±0.97 versus 86.01±0.94; Claude Sonnet is 86.52±1.21 versus 86.78±1.17; Llama 3.1-405B is 86.72±1.95 versus 87.44±1.29; and Mistral Small is 85.72±1.71 versus 87.24±1.75. These discrepancies are too large to be dismissed as typographical noise and they directly affect the ranking analysis and the conclusion. The authors must reconcile the values in the table and the figure, and should state which numbers are authoritative. Without this, the headline claim is not stable.
  3. [Sections 3.3.2, 4.2, and 6] Correctness is operationalized solely as BERTScore against the expert ground truth, with no human expert evaluation. Section 6 itself acknowledges the absence of human evaluation, and the prior validation of BERTScore for CNG cited in Section 2 (Giorgi et al., 2023) uses a different setup with in-context learning and temperature 0.2. Because BERTScore can be insensitive to clinically meaningful errors, such as a missing medication or an incorrect dosage, the statement that generated notes are 'correct' or 'close to expert notes' is not yet established. At minimum, the authors should specify the HuggingFace checkpoint used, and provide a calibration of BERTScore against clinician judgment or entity-level accuracy on a subset of the data. This is load-bearing because correctness is one of the two axes on which the 'most reliable' ranking is built.
minor comments (5)
  1. [Section 3.2.1 and Appendix C] The prompt template for Llama models is modified to include the special Llama chat-format tokens. Please clarify whether the change is purely the required chat template or whether any other wording was altered; otherwise the cross-family comparison is confounded and the apparent superiority of Llama 70B over proprietary models may be an artifact of the prompt format.
  2. [Section 4.1] The claim that intra-prompt stability 'may be measured using semantic consistency alone' is stronger than the evidence supports. Consistency rate and semantic consistency capture different aspects of reliability, and the paper itself notes the value of string equivalence as a strict measure. I suggest softening this claim or providing a quantitative argument for why CR can be dropped.
  3. [Table 1] There are minor typographical issues in Table 1: 'ChatGPT-40-mini' should be 'ChatGPT-4o-mini', and the use of 'LLama' in Figure 3 should be made consistent with 'Llama' elsewhere.
  4. [Section 1 and 6] The statement that 'no study exploring the reliability of open-weight LLMs in CNG was found' is a strong claim; please provide a brief summary of the search strategy or soften the claim to avoid overstating the novelty.
  5. [Reproducibility] The manuscript does not include a data or code availability statement. Given the reproducibility value of the experimental protocol (112 transcripts, 10 iterations, explicit model configurations), adding a link to the generation and evaluation code would be useful.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the reliability metrics are direct measurements against an external benchmark, with no fitted parameter or self-citation chain used as a prediction.

full rationale

The paper's derivation chain is: take aci-bench transcripts and expert ground-truth notes, prompt each LLM ten times with the same template, compute consistency rate (string equality), semantic consistency (BERTScore among the ten generated notes), and semantic similarity (BERTScore against the expert note), then aggregate by medians. Each step is an independent measurement of model output against a fixed external dataset. No parameter is fit from the outcome, no uniqueness theorem is imported from the authors, and no reported score is an input to itself by construction. Using BERTScore for both SC and SS is a shared metric choice, but not a reduction: SC compares generated-generated pairs while SS compares generated-ground-truth pairs, so the two quantities are not identical by definition and can move independently. The statement that Llama 70B and Mistral Small are 'most reliable' is an informal joint reading of SC and SS without a specified aggregation function or significance test; that is an under-specified analysis rather than circular reasoning. The numerical discrepancies between Table 2 and Figure 3 and the overlapping IQRs weaken confidence in the ranking, but they are empirical consistency problems, not circularity. The Limitations section explicitly acknowledges the absence of human expert evaluation, prompt optimization, and clinical validation, which is a validity limitation rather than a circular step. Prior work is cited for metric suitability and comparison, and the authors' own SINTA system is mentioned only as motivation, not as evidence for any result. No load-bearing step is justified solely by a self-citation, and no claim is equivalent to its inputs by definition. Therefore the appropriate circularity score is 0.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The paper introduces no fitted parameters and no invented entities. The measurements depend on three domain assumptions: BERTScore meaningfully captures semantic equivalence and correctness for clinical notes, the aci-bench ground truth notes represent expert quality, and the API configuration with temperature 0 plus top_p 0 plus top_k 1 yields near-deterministic outputs. These assumptions are stated or standard within the paper, but each carries some epistemic risk.

assumptions (3)
  • domain assumption BERTScore is a valid measure of semantic consistency and correctness for clinical notes.
    The paper uses BERTScore to define both semantic consistency and semantic similarity (Sections 2 and 3.3.2). Prior work cited by the authors (Giorgi et al. 2023; Moramarco et al. 2022) supports BERTScore for CNG, but no human validation is performed in this study, and the paper's own limitations call for expert evaluation.
  • domain assumption The aci-bench ground truth notes are expert-quality gold standards.
    Correctness is measured by comparing generated notes to the dataset's ground truth notes (Section 3.1). If these reference notes are not representative of real expert charting, the semantic similarity scores understate or overstate clinical correctness.
  • domain assumption Setting temperature=0, top_p=0, and top_k=1 (where supported) maximizes determinism across API-served models.
    The method section states parameters were set to maximize determinism (Section 3.2.2), yet API serving can introduce nondeterminism, and the paper provides no seeds or API-specific versioning. This affects the interpretation of consistency rates.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Are LLMs reliable? An exploration of the reliability of large language models in clinical note generation." pith.science (2026). https://pith.science/paper/GQW7BTG2

@misc{pith2026250517095,
  author       = {Pith},
  title        = {Pith review of: Are LLMs reliable? An exploration of the reliability of large language models in clinical note generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GQW7BTG2}},
  note         = {Machine review of arXiv:2505.17095}
}
read the original abstract

Due to the legal and ethical responsibilities of healthcare providers (HCPs) for accurate documentation and protection of patient data privacy, the natural variability in the responses of large language models (LLMs) presents challenges for incorporating clinical note generation (CNG) systems, driven by LLMs, into real-world clinical processes. The complexity is further amplified by the detailed nature of texts in CNG. To enhance the confidence of HCPs in tools powered by LLMs, this study evaluates the reliability of 12 open-weight and proprietary LLMs from Anthropic, Meta, Mistral, and OpenAI in CNG in terms of their ability to generate notes that are string equivalent (consistency rate), have the same meaning (semantic consistency) and are correct (semantic similarity), across several iterations using the same prompt. The results show that (1) LLMs from all model families are stable, such that their responses are semantically consistent despite being written in various ways, and (2) most of the LLMs generated notes close to the corresponding notes made by experts. Overall, Meta's Llama 70B was the most reliable, followed by Mistral's Small model. With these findings, we recommend the local deployment of these relatively smaller open-weight models for CNG to ensure compliance with data privacy regulations, as well as to improve the efficiency of HCPs in clinical documentation.

Figures

Figures reproduced from arXiv: 2505.17095 by the authors.

Figure 1
Figure 1. Large Language Model (LLM) Reliability Evaluation Framework for the Task of Clinical Note Generation (CNG). This has two phases, CNG and reliability evaluation, which are executed for each transcript that has a corresponding clinical note made by an expert (ground truth note). (1) CNG starts with said transcript being incorporated into a user prompt template, which then serves as an input to an LLM. The LLM response… view at source ↗
Figure 3
Figure 3. Correctness of LLMs in CNG. Except for Mistral, the larger models per model family performed better than their smaller models. the performance of Meta’s Llama 70B model and Mistral’s Small model, which both had perfect se￾mantic consistency despite having an imperfect, but notably high, consistency rate. In general, all models had a semantic consistency greater than 96% regardless of the consistency rate, which vari… view at source ↗
Figure 2
Figure 2. Intra-prompt stability of LLMs in CNG. Despite LLMs generating notes written in varied ways, the meaning of these notes were relatively consistent across multiple iterations, implying that these LLMs performed well in terms of intra-prompt stability. choice question-answering tasks. Although the out￾puts of such tasks were linguistically controllable and are short-form texts, none of the LLMs studied by Atil et al. … view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Comparison of intra-prompt stability and correctness of LLMs in CNG. Meta’s Llama 70B model and Mistral’s Small model appear to be among the most reliable models. 4.3 Overall LLM Reliability Shown in [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Sample data from the aci-bench dataset. An example of the corrected transcript of a natural conversation between a patient and a doctor (clinical conversation transcript), together with its corresponding clinical note which serves as the ground truth note for this stud…
Figure 6
Figure 6. Figure 6: User Prompt Template. This was used to keep the format consistent across all models. <|begin_of_text|> <|start_header_id|>user<|end_header_id|> user_prompt <|eot_id|> [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]
Figure 7
Figure 7. Figure 7: Formatted Prompt Template. This was used to keep the format consistent across all Llama models. user_prompt here refers to the input which contains the transcript included in the User Prompt Template ( [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

35 extracted references · 20 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Berk Atil, Alexa Chittams, Liseng Fu, Ferhan Ture, Lixinyu Xu, and Breck Baldwin. 2024. http://arxiv.org/abs/2408.04667 LLM Stability : A detailed analysis with some surprises . arXiv preprint. ArXiv:2408.04667 [cs]

    arXiv 2024

  4. [4]

    Rahmani, and Youlin Li

    Iman Azimi, Mohan Qi, Li Wang, Amir M. Rahmani, and Youlin Li. 2025. https://doi.org/10.1038/s41598-024-85003-w Evaluation of LLMs accuracy and consistency in the registered dietitian exam through prompt engineering and knowledge retrieval . Scientific Reports, 15(1):1506. Publisher: Nature Publishing Group

    work page doi:10.1038/s41598-024-85003-w 2025

  5. [5]

    Sebire, Saleh Khalil, Elham Asgari, Christopher Tan, Andrew Taylor, and Dominic Pimenta

    Jasmine Balloch, Shankar Sridharan, Geralyn Oldham, Jo Wray, Paul Gough, Robert Robinson, Neil J. Sebire, Saleh Khalil, Elham Asgari, Christopher Tan, Andrew Taylor, and Dominic Pimenta. 2024. https://doi.org/10.1016/j.fhj.2024.100157 Use of an ambient artificial intelligence tool to improve quality of clinical documentation . Future Healthcare Journal, 1...

    arXiv 2024

  6. [6]

    Christopher Barrie, Elli Palaiologou, and Petter Törnberg. 2024. https://doi.org/10.48550/arXiv.2407.02039 Prompt Stability Scoring for Text Annotation with Large Language Models . arXiv preprint. ArXiv:2407.02039 [cs]

    arXiv doi:10.48550/arxiv.2407.02039 2024

  7. [7]

    Anjanava Biswas and Wrick Talukdar. 2024. http://arxiv.org/abs/2405.18346 Intelligent Clinical Documentation : Harnessing Generative AI for Patient - Centric Clinical Note Generation . arXiv preprint. ArXiv:2405.18346

    arXiv 2024

  8. [8]

    Yu-Wen Chen and Julia Hirschberg. 2024. https://doi.org/10.18653/v1/2024.clinicalnlp-1.1 Exploring Robustness in Doctor - Patient Conversation Summarization : An Analysis of Out -of- Domain SOAP Notes . In Proceedings of the 6th Clinical Natural Language Processing Workshop , pages 1--9, Mexico City, Mexico. Association for Computational Linguistics

    work page doi:10.18653/v1/2024.clinicalnlp-1.1 2024

Show all 35 references
  1. [9]

    Furui Cheng, Vilém Zouhar, Simran Arora, Mrinmaya Sachan, Hendrik Strobelt, and Mennatallah El-Assady. 2024. https://doi.org/10.1145/3613904.3641904 RELIC : Investigating Large Language Model Responses using Self - Consistency . In Proceedings of the 2024 CHI Conference on Hum...

    2024

  2. [10]

    Vittoria Dentella, Fritz Günther, and Evelina Leivada. 2023. https://doi.org/10.1073/pnas.2309583120 Systematic testing of three Language Models reveals low language accuracy, absence of response stability, and a yes-response bias . Proceedings of the National Academy of Scien...

    2023 doi

  3. [11]

    Nambudiri

    Shilpa Ghatnekar, Adam Faletsky, and Vinod E. Nambudiri. 2021. https://doi.org/10.1007/s12553-021-00568-0 Digital scribe utility and barriers to implementation in clinical practice: a scoping review . Health and Technology, 11(4):803--809

    2021 doi

  4. [12]

    John Giorgi, Augustin Toma, Ronald Xie, Sondra Chen, Kevin An, Grace Zheng, and Bo Wang. 2023. https://doi.org/10.18653/v1/2023.clinicalnlp-1.36 W ang L ab at MEDIQA -chat 2023: Clinical note generation from doctor-patient conversations using large language models . In Proceed...

    2023 doi

  5. [13]

    Felix Heilmeyer, Daniel B \"o hringer, Thomas Reinhard, Sebastian Arens, Lisa Lyssenko, and Christian Haverkamp. 2024. https://doi.org/10.2196/59617 Viability of open large language models for clinical documentation in german health care: Real-world model evaluation study . JM...

    2024 doi

  6. [14]

    Gold, and Vishnu Mohan

    Annessa Kernberg, Jeffrey A. Gold, and Vishnu Mohan. 2024. https://doi.org/10.2196/54419 Using ChatGPT -4 to Create Structured Medical Notes From Audio Recordings of Physician - Patient Encounters : Comparative Study . Journal of Medical Internet Research, 26(1):e54419. Compan...

    2024 doi

  7. [15]

    Akdogan, Jessica Atkins, Mohamed B

    Elie Kozaily, Mabelissa Geagea, Ecem R. Akdogan, Jessica Atkins, Mohamed B. Elshazly, Maya Guglin, Ryan J. Tedford, and Ramsey M. Wehbe. 2024. https://doi.org/10.1016/j.ijcard.2024.132115 Accuracy and consistency of online large language model-based artificial intelligence cha...

    2024

  8. [16]

    Taiji Li, Zhi Li, and Yin Zhang. 2024. https://aclanthology.org/2024.lrec-main.771/ Improving Faithfulness of Large Language Models in Summarization via Sliding Generation and Self - Consistency . In Proceedings of the 2024 Joint International Conference on Computational Lingu...

    2024

  9. [17]

    Zheheng Luo, Qianqian Xie, and Sophia Ananiadou. 2024. https://doi.org/10.1016/j.eswa.2024.124456 Factual consistency evaluation of summarization in the Era of large language models . Expert Systems with Applications, 254:124456

    2024

  10. [18]

    Lientje Maas, Mathan Geurtsen, Florian Nouwt, Stefan Schouten, Robin van de Water, Sandra van Dulmen, Fabiano Dalpiaz, Kees van Deemter, and Sjaak Brinkkemper. 2020. https://core.ac.uk/reader/286030500 The Care2Report System : Automated Medical Reporting as an Integrated Solut...

    2020

  11. [19]

    McCoy, Faye Yu Ci Ng, Christopher M

    Liam G. McCoy, Faye Yu Ci Ng, Christopher M. Sauer, Katelyn Edelwina Yap Legaspi, Bhav Jain, Jack Gallifant, Michael McClurkin, Alessandro Hammond, Deirdre Goode, Judy Gichoya, and Leo Anthony Celi. 2024. https://doi.org/10.1186/s12909-024-06048-z Understanding and training fo...

    2024 doi

  12. [20]

    Pennathur

    Amirmasoud Momenipour and Priyadarshini R. Pennathur. 2019. https://doi.org/10.1016/j.ergon.2019.06.012 Balancing documentation and direct patient care activities: A study of a mature electronic health record system . International Journal of Industrial Ergonomics, 72:338--346

    2019 doi

  13. [21]

    Francesco Moramarco, Alex Papadopoulos Korfiatis, Mark Perera, Damir Juric, Jack Flann, Ehud Reiter, Anya Belz, and Aleksandar Savkov. 2022. https://doi.org/10.18653/v1/2022.acl-long.394 Human evaluation and correlation with automatic metrics in consultation note generation . ...

    2022 doi

  14. [22]

    Quiroz, Liliana Laranjo, Ahmet Baki Kocaballi, Shlomo Berkovsky, Dana Rezazadegan, and Enrico Coiera

    Juan C. Quiroz, Liliana Laranjo, Ahmet Baki Kocaballi, Shlomo Berkovsky, Dana Rezazadegan, and Enrico Coiera. 2019. https://doi.org/10.1038/s41746-019-0190-1 Challenges of developing a digital scribe to reduce clinical documentation burden . npj Digital Medicine, 2(1):1--6. Nu...

    2019 doi

  15. [23]

    Thomas Savage, John Wang, Robert Gallo, Abdessalem Boukil, Vishwesh Patel, Seyed Amir Ahmad Safavi-Naini, Ali Soroush, and Jonathan H. Chen. 2024. https://doi.org/10.1093/jamia/ocae254 Large language model uncertainty proxies: discrimination and calibration for medical diagnos...

    2024 doi

  16. [24]

    Yash Saxena, Sarthak Chopra, and Arunendra Mani Tripathi. 2024. https://doi.org/10.48550/arXiv.2404.16478 Evaluating Consistency and Reasoning Capabilities of Large Language Models . arXiv preprint. ArXiv:2404.16478 [cs]

    2024 doi arXiv

  17. [25]

    Victoria Tucci, Joan Saary, and Thomas E. Doyle. 2021. https://jmai.amegroups.org/article/view/6664 Factors influencing trust in medical artificial intelligence for healthcare professionals: a narrative review . Journal of Medical Artificial Intelligence, 5(0)

    2021

  18. [26]

    Joshua Yi Min Tung, Sunil Ravinder Gill, Gerald Gui Ren Sng, Daniel Yan Zheng Lim, Yuhe Ke, Ting Fang Tan, Liyuan Jin, Kabilan Elangovan, Jasmine Chiat Ling Ong, Hairil Rizal Abdullah, Daniel Shu Wei Ting, and Tsung Wen Chong. 2024. https://doi.org/10.2196/57721 Comparison of ...

    2024 doi

  19. [27]

    Hanyin Wang, Chufan Gao, Bolun Liu, Qiping Xu, Guleid Hussein, Mohamad El Labban, Kingsley Iheasirim, Hariprasad Korsapati, Chuck Outcalt, and Jimeng Sun. 2024 a . https://arxiv.org/abs/2405.00715 Adapting open-source large language models for cost-effective, expert-level clin...

    2024 arXiv

  20. [28]

    Leyao Wang, Zhiyu Wan, Congning Ni, Qingyuan Song, Yang Li, Ellen Clayton, Bradley Malin, and Zhijun Yin. 2024 b . https://doi.org/10.2196/22769 Applications and Concerns of ChatGPT and Other Conversational Large Language Models in Health Care : Systematic Review . Journal of ...

    2024 doi

  21. [29]

    Li Wang, Xi Chen, XiangWen Deng, Hao Wen, MingKe You, WeiZhi Liu, Qi Li, and Jian Li. 2024 c . https://doi.org/10.1038/s41746-024-01029-4 Prompt engineering in consistency and reliability with the evidence-based guideline for LLMs . npj Digital Medicine, 7(1):1--9. Publisher: ...

    2024 doi

  22. [30]

    Yuxuan Wu, Mingyue Wu, Changyu Wang, Jie Lin, Jialin Liu, and Siru Liu. 2024. https://doi.org/10.2196/54811 Evaluating the Prevalence of Burnout Among Health Care Professionals Related to Electronic Health Record Use : Systematic Review and Meta - Analysis . JMIR Medical Infor...

    2024 doi

  23. [31]

    Wen-wai Yim, Yujuan Fu, Asma Ben Abacha, Neal Snider, Thomas Lin, and Meliha Yetisgen. 2023. https://doi.org/10.1038/s41597-023-02487-3 Aci-bench: a Novel Ambient Clinical Intelligence Dataset for Benchmarking Automatic Visit Note Generation . Scientific Data, 10(1):586. Publi...

    2023 doi

  24. [32]

    Wen-wai Yim, Yujuan Fu, Asma Ben Abacha, and Meliha Yetisgen. 2024. https://aclanthology.org/2024.lrec-main.1409 To Err Is Human , How about Medical Large Language Models ? Comparing Pre -trained Language Models for Medical Assessment Errors and Reliability . In Proceedings of...

    2024

  25. [33]

    Weinberger, and Yoav Artzi

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. https://arxiv.org/abs/1904.09675 Bertscore: Evaluating text generation with bert . Preprint, arXiv:1904.09675

    2020 arXiv

  26. [34]

    Zhan Zhang, Karen Joy, Richard Harris, and Sun Young Park. 2022. https://doi.org/10.1145/3555111 Characteristics and Challenges of Clinical Documentation in Self - Organized Fast - Paced Medical Work . Proceedings of the ACM on Human-Computer Interaction, 6(CSCW2):386:1--386:21

    2022 doi

  27. [35]

    Yukun Zhao, Lingyong Yan, Weiwei Sun, Guoliang Xing, Shuaiqiang Wang, Chong Meng, Zhicong Cheng, Zhaochun Ren, and Dawei Yin. 2024. https://aclanthology.org/2024.lrec-main.782 Improving the Robustness of Large Language Models via Consistency Alignment . In Proceedings of the 2...

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.