Pith. sign in

REVIEW 4 major objections 6 minor 2 cited by

A deterministic, LLM-free fact-checker can catch most unsupported claims in AI-generated clinical summaries, the paper argues, and a domain-tuned generator reduces how many claims are wrong to begin with.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 15:37 UTC pith:DFFBKNCS

load-bearing objection A plausible proposition-level fact-checker with a consistent confusion matrix, but the SOTA claim conflates accuracy with F1 and extraction recall is never measured. the 4 major comments →

arxiv 2512.16189 v4 pith:DFFBKNCS submitted 2025-12-18 cs.CL

Mitigating hallucinations in healthcare LLMs with granular fact-checking and domain-specific adaptation

Musarrat Zeba , Abdullah Al Mamun , Kishoar Jahan Tithee , Debopom Sutradhar , Mohaimenul Azam Khan Raiaan , Saddam Mukta , Reem E. Mohamed , Md Rafiqul Islam
show 3 more authors
Yakub Sebastian Mukhtar Hussain Sami Azam
This is my paper
classification cs.CL
keywords hallucination mitigationclinical summarizationfact-checkingMIMIC-IIIproposition-level verificationLoRA fine-tuningelectronic health records
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that hallucinations in LLM-generated clinical summaries can be caught by a deterministic, LLM-free verifier that decomposes both the summary and the patient's electronic health record into atomic propositions—(entity, attribute, value, time) tuples—and checks each one with discrete logical tests. On 3,786 propositions from 104 MIMIC-III discharge summaries, the verifier reports precision 0.8904, recall 0.8234, and F1 0.8556. Paired with a LoRA-fine-tuned LLaMA-3.1-8B summarizer (ROUGE-1 0.5797, BERTScore 0.9120), the system is meant to make AI-generated summaries trustworthy enough for clinical sign-off. A sympathetic reader would care because the checker is transparent, auditable, and needs no second LLM, though the paper's own §6 concedes it is less reliable in rare oncologic, pediatric metabolic, and transplant ICU cases.

Core claim

The central claim is that factual consistency between a generated clinical summary and its source EHR can be verified at the granularity of individual propositions, without resorting to another generative model. The paper claims its fact-checking module reaches precision 0.8904, recall 0.8234, and F1 0.8556, and that the same pipeline outperforms previous systems on MIMIC-III claim verification, exceeding a reported 79.13% accuracy. The generator component, fine-tuned with Low-Rank Adaptation on the full MIMIC-III training set, produces summaries with lexical and semantic quality competitive with prior clinical summarizers; the authors' stated contribution is the separation of a fluent gener

What carries the argument

The load-bearing object is the proposition tuple p=(e,a,v,t)—clinical entity, attribute, value, time. Both the generated summary and the EHR are decomposed into such tuples by a deterministic pipeline: clinical named-entity recognition, BioPortal concept normalization, dependency-pattern attribute inference, and numeric/temporal extraction. Matching uses cosine similarity of BioClinicalBERT and Sentence-BERT embeddings; then a set of discrete checks (negation, implication, temporal order, mutual exclusivity, numerical consistency, and a presence test for omissions) assigns a Supported/Not-Supported verdict per proposition.

Load-bearing premise

The load-bearing premise is that the deterministic rule-based extractor converts every clinically important fact in both the EHR and the generated summary into a complete, correctly matched (entity, attribute, value, time) proposition; if extraction or matching silently drops a fact, the checker certifies a summary whose omission cannot be seen.

What would settle it

Inject known errors (changed dosages, swapped event order, negated diagnosis, deleted comorbidity) into otherwise correct MIMIC-III summaries, run the verifier, and compare the detection rate to the reported 0.82 recall; also recompute F1 from the full confusion matrix, since Table 3 reports F1=0.8556 while §5.3 calls 79.13% (the accuracy) the state-of-the-art F1.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Any LLM's clinical summary can be passed through the verifier as a post-hoc safety layer, with each flagged proposition carrying a transparent failure label such as numerical mismatch, negation conflict, or missing implied treatment.
  • Clinicians could focus their review on the 'Not Supported' propositions instead of rereading entire summaries, which would reduce the burden of checking AI-generated text.
  • The LoRA-fine-tuned LLaMA-3.1-8B generator demonstrates that parameter-efficient adaptation on about 26,000 discharge summaries yields clinically coherent text, so resource-constrained teams can build domain-specific summarizers.
  • The verifier's deterministic operation makes results reproducible and auditable in privacy-sensitive deployments, without the model drift or opacity of an LLM-as-judge.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The verifier's Supported/Not-Supported labels could be reused as a reward signal for further fine-tuning, potentially pushing hallucination rates below what the paper reports; the paper does not explore this closed loop.
  • Because the checker's recall (0.82) is bounded by the recall of the rule-based proposition extractor, a direct measurement of extraction completeness—how many gold EHR facts survive the (e,a,v,t) conversion—would reveal whether remaining hallucinations are verification failures or extraction failures.
  • The paper's §5.3 calls 79.13% the 'state-of-the-art F1', but that value is the accuracy reported in Table 3 (F1 = 0.8556); a metric-consistent re-benchmark against Claude-2+DOSSIER would clarify whether the claimed SOTA holds.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes a two-stage system for clinical summarization: a LoRA fine-tuned LLaMA-3.1-8B generator trained on MIMIC-III discharge summaries, and a separate, non-generative fact-checking module that decomposes both summaries and EHRs into (entity, attribute, value, time) propositions and applies semantic, numerical, temporal, logical, and presence checks to assign 'Supported'/'Not Supported' verdicts. On 3,786 propositions from 104 summaries, the fact-checking module reports precision 0.8904, recall 0.8234, F1 0.8556, and accuracy 0.7913; the summarization model reports ROUGE-1 0.5797 and BERTScore 0.9120. The paper further claims a new state-of-the-art F1-score of 79.13% on MIMIC-III, surpassing Claude-2+DOSSIER. The central contribution is a transparent, LLM-free, proposition-level verification layer that can be applied post hoc to any clinical summarizer.

Significance. If the reported numbers reflect true clinical factuality, the deterministic verifier would be a valuable contribution: it is transparent, auditable, does not rely on LLM-as-judge, and explicitly targets fine-grained errors (dosages, times, negations, omissions) that sentence- or PICO-level evaluations miss. The proposition-level formalism and the separation of generation from verification are conceptually sound, and the confusion-matrix arithmetic is internally consistent. The main strengths are the deterministic rule-based checks, the emphasis on EHR grounding, and the explicit acknowledgment of coverage limits in specialized domains. However, the headline evaluation depends critically on the completeness and correctness of the rule-based extractor, which is not independently measured; this limits what can be concluded from the current numbers.

major comments (4)
  1. [§3.2.1, Eqs. (5)–(6); §5.2] The 3,786-proposition evaluation set is the output of the same rule-based extractor used by the verifier. The paper never measures the extractor's recall against an independently clinician-authored proposition list, so a hallucination in a proposition the extractor silently drops—e.g., a rare oncologic or transplant-related fact, as §6 concedes—is invisible to both the system and the reported metrics. The headline recall of 0.8234 therefore measures recall over extracted summary propositions, not over clinically important claims. Please report extraction recall, or at least bound its effect, e.g., by having clinicians enumerate gold propositions on a subsample and recomputing precision/recall over that set.
  2. [§5.3, Table 5; §5.2, Table 3] The claim of a 'new state-of-the-art F1-score of 79.13%' is a mislabeling: Table 3 reports F1=0.8556 and accuracy=0.7913, and the 79.13% figure is the accuracy. Moreover, Table 5 compares accuracy values taken from other papers (e.g., DOSSIER) without verifying that those systems were evaluated on the same proposition-extraction protocol, same label set, or same 104 summaries. Cross-paper accuracy comparisons are not meaningful unless the evaluation setups are identical. Please clarify that the SOTA claim is about accuracy, not F1, and either reproduce baselines under the same protocol or qualify the comparison.
  3. [§4.7, §6] The gold-standard verdicts were provided by only two clinicians, and the paper reports no inter-annotator agreement, no blinding to the system's outputs, and no specification of how many summaries were reviewed ('a random subset'). The 'roughly 85% clinically acceptable' figure appears only in the Discussion, not in the Results. Since the verifier's precision/recall are measured against these human labels, label noise or confirmation bias could materially affect the reported 0.8904/0.8234. Please report the annotation protocol, agreement statistics, and the exact size of the reviewed subset.
  4. [§3.2.3, Eqs. (9)–(14)] The logical checks (negation, implication, temporal, exclusivity, numerical, presence) depend on hand-authored clinical rule sets and on the assumption that extraction is complete. No evaluation is reported for the coverage or accuracy of these rules themselves; for example, the implication rule 'pneumonia ⇒ antibiotics' may be clinically valid in many contexts but not all, and a false implication would systematically mark supported propositions as unsupported (or vice versa). Please provide the rule inventory, the number of rules per check, and a manual audit of rule false-positive/negative rates on a sample.
minor comments (6)
  1. [Abstract / §3.1] The method is inconsistently spelled as 'LoRa' in the abstract and 'LoRA' elsewhere; please standardize.
  2. [§4.2] 'a slight moralization was performed' should likely read 'normalization' or 'text normalization'; as written it is unclear.
  3. [§5.3, Table 5] The text states 'CodeLlama-13B [57] ... 65.76%' but Table 5 lists CodeLlama-13B at 64.13% and CodeLlama-13B (DOSSIER) at 65.76%. The in-text and table values are inconsistent.
  4. [§5.2] The paragraph beginning 'Table 4 reports additional evaluation metrics...' appears twice nearly verbatim.
  5. [Throughout] Several table and figure captions begin with stray spaces ('T able 2', 'T able 3'), and some references are incomplete or duplicated (e.g., [48] appears to duplicate [22]).
  6. [§3.2.3, Eq. (11)] The temporal check uses t^i_S < t^j_S, but the notation for which proposition's time marker is used is not defined consistently; clarify that i and j refer to the matched summary and EHR propositions.

Circularity Check

1 steps flagged

Fact-checker metrics are computed on the proposition set generated by the module's own extractor, so recall excludes every claim the extractor misses; clinician 'gold' labels are corrections of the module's findings on that same set.

specific steps
  1. self definitional [Abstract; §3.2.1 Eqs. (5)–(6); §4.6; §5.2]
    "For evaluation of the fact-checking module, we sampled 104 summaries, extracted them into 3786 propositions, and used these as facts. ... In practice, we extract (e, a, v, t) from both the generated summary S and the corresponding EHR document E using a deterministic rule-based pipeline. ... each extracted proposition was logically compared using the proposed fact-checking module with its corresponding EHR proposition and labeled as either ”Supported” or ”Not Supported.” Then a clinician reviewed and checked these findings to make sure that there were no inaccuracies in the results."

    The 3,786-proposition evaluation set is produced by the same rule-based extractor (Eq. 6) that feeds the fact-checker. Every verdict and all precision/recall/F1 counts therefore range only over propositions the extractor happens to surface; a hallucinated or omitted summary claim that the extractor drops never enters PS or PE, so it cannot be counted as a false negative. Presence check Eq. (14) can similarly flag omissions only among extracted EHR propositions. Thus the reported recall (0.8234) is conditional recall over extractor-emitted propositions, not recall over clinically important claims as §5.2 claims ('most of the clinically valid facts in the summaries are found and verified'). The clinician gold labels are also corrections of the module's own findings on the same extracted prop

full rationale

The paper's central fact-checking metrics are partially circular: the evaluation universe is defined by the module's own proposition extractor, so any unsupported claim that the extractor misses is invisible to both the module and the reported recall. This is not a case of fitted parameters renamed as predictions, nor is it driven by self-citation; the LoRA formulation is standard and cited to Hu et al., and the DOSSIER comparison is to external prior work. The module's decisions on extracted propositions are not predetermined — clinician review provides some external signal — so the circularity is partial rather than total. The §6 concession that the pipeline is unreliable for rare oncologic subtypes, pediatric metabolic disorders, and transplant/ICU scenarios corroborates the extraction blind spot. A further consistency issue, noted for completeness but not treated as circularity, is that §5.3 calls 79.13% a 'state-of-the-art F1-score' while Table 3 reports accuracy 0.7913 and F1 0.8556.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 0 invented entities

The checker's verdicts are determined by what the paper supplies by hand: the proposition schema and extractor rules (§3.2.1), the implication/exclusivity/negation rule sets (§3.2.3), and an unreported similarity threshold. The clinician gold labels (n=2, no IAA) are the external anchor, but their independence from the proposition scheme is not established. No new physical or categorical entities are introduced; the (e,a,v,t) tuple is a representational device.

free parameters (3)
  • Cosine-similarity matching threshold
    Verdicts depend on a similarity cutoff to declare an EHR counterpart 'found' (Eqs. 7-8, presence check Eq. 14). The threshold value is never reported; if tuned on the same 104-summary sample, it would inflate the headline F1.
  • Hand-authored clinical rule sets
    Negation cues, implication rules (e.g., pneumonia⇒antibiotics), mutual-exclusivity pairs, and temporal normalizations (§3.2.3) are written by the authors and directly determine Not-Supported verdicts; their coverage drives precision/recall.
  • LoRA hyperparameters = r=8, lr=1e-4, α not stated
    Rank, scaling α, and learning rate chosen by hand (§3.1.1, §4.3); they affect summarization metrics but not the fact-checker's core numbers.
axioms (5)
  • domain assumption Every clinically important fact in summary and EHR is representable as a complete (e, a, v, t) proposition captured without loss by the rule-based extractor.
    Invoked in §3.2.1 (Eqs. 5-6). The paper's own §6 concedes rare oncologic, pediatric, and transplant scenarios are under-covered, so the assumption fails exactly where errors matter most.
  • domain assumption BioClinicalBERT/Sentence-BERT cosine similarity selects the true EHR counterpart for each summary proposition.
    §3.2.2 (Eqs. 7-8). A wrong match routes all downstream checks onto the wrong reference; no matching-error analysis is given.
  • ad hoc to paper The hand-written logical rules encode correct clinical constraints.
    §3.2.3 (Eqs. 9-13). Rules such as 'pneumonia⇒antibiotics' and the exclusivity pairs are introduced here, not derived from a clinical standard; a missing or wrong rule silently flips verdicts.
  • domain assumption Clinician labels on the 3,786 propositions are accurate ground truth.
    §4.7 and §6 report n=2 reviewers with no inter-annotator agreement or blinding protocol.
  • standard math LoRA low-rank update (W' = W + (α/r)AB^T) and the NLL objective suffice for domain adaptation.
    Eqs. 1-4; standard background from Hu et al. [24], not in dispute.

pith-pipeline@v1.3.0-alltime-deepseek · 22608 in / 24052 out tokens · 201856 ms · 2026-08-03T15:37:27.879197+00:00 · methodology

0 comments
read the original abstract

In healthcare, it is essential for any Large Language Model (LLM)-generated output to be reliable and accurate, particularly in cases involving decision-making and patient safety. However, the outputs are often unreliable in such critical areas due to the risk of hallucinated outputs from the LLMs. To address this issue, we propose a fact-checking module that operates independently of any LLM, along with a domain-specific summarization model designed to minimize hallucination rates. Our model is fine-tuned using Low-Rank Adaptation (LoRA) on the MIMIC-III dataset and is paired with the fact-checking module, which uses numerical tests for correctness and logical checks at a granular level through discrete logic in natural language processing (NLP) to validate facts against electronic health records (EHRs). We trained the LLM on the full MIMIC-III dataset. For evaluation of the fact-checking module, we sampled 104 summaries, extracted them into 3786 propositions, and used these as facts. The fact-checking module achieves a precision of 0.8904, a recall of 0.8234, and an F1-score of 0.8556. Additionally, the LLM summary achieves a ROUGE-1 score of 0.5797 and a BERTScore of 0.9120 for summary quality.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Serialisation Strategy Matters: How FHIR Data Format Affects LLM Medication Reconciliation

    cs.CL 2026-04 conditional novelty 7.0

    Clinical narrative format beats raw JSON for LLMs up to 8B parameters on medication reconciliation but raw JSON wins at 70B scale, with omissions as the main error type.

  2. Interpretable Language Model for Closed-Loop Type 1 Diabetes Control

    cs.AI 2026-06 reject novelty 5.0

    LLM-T1D fine-tunes LLaMA/Qwen to mimic an RL insulin-dosing expert and claims better glucose control, but the reported numbers are synthesized and unreproducible.

Reference graph

Works this paper leans on

63 extracted references · 21 linked inside Pith · cited by 2 Pith papers

  1. [1]

    Roles and potential of large language models in healthcare: A comprehensive review,

    C. Lin and C.-F. Kuo, “Roles and potential of large language models in healthcare: A comprehensive review,”Biomedical Journal, p. 100868, 2025. [Online]. Available: https://www.sciencedirect.com/ science/article/pii/S2319417025000423

    2025

  2. [2]

    Bioknowprompt: Incorporating imprecise knowledge into prompt-tuning verbalizer with biomedical text for relation extraction,

    Q. Li, Y. Wang, T. You, and Y. Lu, “Bioknowprompt: Incorporating imprecise knowledge into prompt-tuning verbalizer with biomedical text for relation extraction,” Information Sciences, vol. 617, pp. 346–358, 2022

    2022

  3. [3]

    Medical hallucinations in foundation models and their impact on healthcare,

    Y. Kim, H. Jeong, S. Chen, S. S. Li, M. Lu, K. Alhamoud, J. Mun, C. Grau, M. Jung, R. Gameiroet al., “Medical hallucinations in foundation models and their impact on healthcare,”arXiv preprint arXiv:2503.05777, 2025

    arXiv 2025

  4. [4]

    Uncertainty-aware multi- criteria decision analysis for evaluation of explainable ar- tificial intelligence methods: A use case from the health- care domain,

    K. Aliyeva and N. Mehdiyev, “Uncertainty-aware multi- criteria decision analysis for evaluation of explainable ar- tificial intelligence methods: A use case from the health- care domain,”Information sciences, vol. 657, p. 119987, 16 2024

    2024

  5. [5]

    Mcd-ears: A multi- modal cross-domain expertise-aware recommender sys- tem for healthcare applications,

    Z. Li, H. Jiang, and S. Zhao, “Mcd-ears: A multi- modal cross-domain expertise-aware recommender sys- tem for healthcare applications,”Information Sciences, p. 122821, 2025

    2025

  6. [6]

    Scene generalization for biomedical fact verifi- cation via hierarchical mixture of experts,

    J. Si, H. Zhu, Y. Zhao, W. Zhang, T. Wang, W. Lu, and D. Zhou, “Scene generalization for biomedical fact verifi- cation via hierarchical mixture of experts,”Information Sciences, p. 122527, 2025

    2025

  7. [7]

    A survey of hal- lucination in large foundation models,

    V. Rawte, A. Sheth, and A. Das, “A survey of hal- lucination in large foundation models,”arXiv preprint arXiv:2309.05922, 2023

    Pith/arXiv arXiv 2023

  8. [8]

    On faithfulness and factuality in abstractive summariza- tion,

    J. Maynez, S. Narayan, B. Bohnet, and R. McDonald, “On faithfulness and factuality in abstractive summariza- tion,”arXiv preprint arXiv:2005.00661, 2020

    Pith/arXiv arXiv 2005

  9. [9]

    Prescrib- ing the right remedy: Mitigating hallucinations in large vision-language models via targeted instruction tuning,

    R. Hu, Y. Tu, S. Wei, D. Lu, and J. Sang, “Prescrib- ing the right remedy: Mitigating hallucinations in large vision-language models via targeted instruction tuning,” Information Sciences, p. 122361, 2025

    2025

  10. [10]

    Virtsi: A novel trust dynamics model enhancing artifi- cial intelligence collaboration with human users–insights from a chatgpt evaluation study,

    M. Virvou, G. A. Tsihrintzis, and E.-A. Tsichrintzi, “Virtsi: A novel trust dynamics model enhancing artifi- cial intelligence collaboration with human users–insights from a chatgpt evaluation study,”Information Sciences, vol. 675, p. 120759, 2024

    2024

  11. [11]

    Med- halt: Medical domain hallucination test for large lan- guage models,

    A. Pal, L. K. Umapathi, and M. Sankarasubbu, “Med- halt: Medical domain hallucination test for large lan- guage models,”arXiv preprint arXiv:2307.15343, 2023

    Pith/arXiv arXiv 2023

  12. [12]

    Generative ai hallucinations in health- care: A challenge for prompt engineering and creativity,

    V. Geroimenko, “Generative ai hallucinations in health- care: A challenge for prompt engineering and creativity,” inHuman-Computer Creativity: Generative AI in Edu- cation, Art, and Healthcare. Springer, 2025, pp. 321– 335

    2025

  13. [13]

    Embracing large lan- guage models for medical applications: opportunities and challenges,

    M. Karabacak and K. Margetis, “Embracing large lan- guage models for medical applications: opportunities and challenges,”Cureus, vol. 15, no. 5, 2023

    2023

  14. [14]

    Large language models encode clinical knowledge,

    K. Singhal, S. Azizi, T. Tu, S. S. Mahdavi, J. Wei, H. W. Chung, N. Scales, A. Tanwani, H. Cole-Lewis, S. Pfohl, P. Payne, M. Seneviratne, P. Gamble, C. Kelly, A. Babiker, N. Sch¨ arli, A. Chowdhery, P. Mansfield, D. Demner-Fushman, B. Ag¨ uera Y Arcas, D. Web- ster, G. S. Corrado, Y. Matias, K. Chou, J. Gottweis, N. Tomasev, Y. Liu, A. Rajkomar, J. Barra...

    2023

  15. [15]

    Clin- icalgpt: Large language models finetuned with diverse medical data and comprehensive evaluation,

    G. Wang, G. Yang, Z. Du, L. Fan, and X. Li, “Clin- icalgpt: Large language models finetuned with diverse medical data and comprehensive evaluation,”arXiv preprint arXiv:2306.09968, 2023. [Online]. Available: https://arxiv.org/abs/2306.09968

    Pith/arXiv arXiv 2023

  16. [16]

    Evaluating large language models on medical evidence summarization,

    L. Tang, Z. Sun, Y. Ma, G. Yang, Y. Gu, V. Yadav, W. Weng, Z. He, Y. Wang, and H. Yu, “Evaluating large language models on medical evidence summarization,”npj Digital Medicine, vol. 6, no. 1, p. 158, 2023. [Online]. Available: https: //www.ncbi.nlm.nih.gov/pmc/articles/PMC10449915/

    2023

  17. [17]

    Opportunities, chal- lenges, and future directions of large language models, including chatgpt in medical education: a systematic scoping review,

    X. Xu, Y. Chen, and J. Miao, “Opportunities, chal- lenges, and future directions of large language models, including chatgpt in medical education: a systematic scoping review,”Journal of educational evaluation for health professions, vol. 21, 2024

    2024

  18. [18]

    Trustworthy ai for medicine: Continuous hallucination detection and elimination with check,

    C. Garcia-Fernandez, L. Felipe, M. Shotande, M. Zitu, A. Tripathi, G. Rasool, I. El Naqa, V. Rudrapatna, and G. Valdes, “Trustworthy ai for medicine: Continuous hallucination detection and elimination with check,” arXiv preprint, 2025. [Online]. Available: https://arxiv. org/abs/2506.11129

    Pith/arXiv arXiv 2025

  19. [19]

    Retrieval- augmented generation for knowledge-intensive nlp tasks,

    P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. K¨ uttler, M. Lewis, W. tau Yih, T. Rockt¨ aschel, S. Riedel, and D. Kiela, “Retrieval- augmented generation for knowledge-intensive nlp tasks,”arXiv preprint arXiv:2005.11401, 2020. [Online]. Available: https://arxiv.org/abs/2005.11401

    Pith/arXiv arXiv 2005

  20. [20]

    Retrieval augmentation reduces hallucination in conversation,

    K. Shuster, S. Poff, M. Chen, D. Kiela, and J. Weston, “Retrieval augmentation reduces hallucination in conversation,” inFindings of the Association for Computational Linguistics: EMNLP

  21. [21]

    Survey of halluci- nation in natural language generation,

    Z. Ji, N. Lee, R. Frieske, T. Yu, D. Su, Y. Xu, E. Ishii, Y. J. Bang, A. Madotto, and P. Fung, “Survey of halluci- nation in natural language generation,”ACM computing surveys, vol. 55, no. 12, pp. 1–38, 2023

    2023

  22. [23]

    Mimic-iii, a freely accessi- ble critical care database,

    A. E. Johnson, T. J. Pollard, L. Shen, L.-w. H. Lehman, M. Feng, M. Ghassemi, B. Moody, P. Szolovits, L. An- thony Celi, and R. G. Mark, “Mimic-iii, a freely accessi- ble critical care database,”Scientific data, vol. 3, no. 1, pp. 1–9, 2016

    2016

  23. [24]

    Lora: Low-rank adaptation of large language models

    E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, W. Chenet al., “Lora: Low-rank adaptation of large language models.”ICLR, vol. 1, no. 2, p. 3, 2022

    2022

  24. [25]

    Factpico: Factuality evaluation for plain language summarization of medical 17 evidence,

    S. A. Joseph, L. Chen, J. Trienes, H. L. G¨ oke, M. Coers, W. Xu, B. C. Wallace, and J. J. Li, “Factpico: Factuality evaluation for plain language summarization of medical 17 evidence,”arXiv preprint arXiv:2402.11456, 2024

    Pith/arXiv arXiv 2024

  25. [26]

    Healthfc: Verifying health claims with evidence-based medical fact- checking,

    J. Vladika, P. Schneider, and F. Matthes, “Healthfc: Verifying health claims with evidence-based medical fact- checking,”arXiv preprint arXiv:2309.08503, 2023

    Pith/arXiv arXiv 2023

  26. [27]

    Minicheck: Efficient fact-checking of llms on grounding documents,

    L. Tang, P. Laban, and G. Durrett, “Minicheck: Efficient fact-checking of llms on grounding documents,”arXiv preprint arXiv:2404.10774, 2024

    Pith/arXiv arXiv 2024

  27. [28]

    Graphcheck: Breaking long-term text barriers with ex- tracted knowledge graph-powered fact-checking,

    Y. Chen, H. Liu, Y. Liu, J. Xie, R. Yang, H. Yuan, Y. Fu, P. Zhou, Q. Chen, J. Caverlee, and I. Li, “Graphcheck: Breaking long-term text barriers with ex- tracted knowledge graph-powered fact-checking,”arXiv preprint arXiv:2502.16514, February 2025

    Pith/arXiv arXiv 2025

  28. [29]

    DOSSIER: Fact checking in electronic health records while preserving patient privacy,

    H. Zhang, S. Nagesh, M. Shyani, and N. Mishra, “DOSSIER: Fact checking in electronic health records while preserving patient privacy,” inProceedings of the 9th Machine Learning for Healthcare Conference, ser. Proceedings of Machine Learning Research, K. Deshpande, M. Fiterau, S. Joshi, Z. Lip- ton, R. Ranganath, and I. Urteaga, Eds., vol

  29. [30]

    Brainllama at semeval-2024 task 6: Prompt- ing llama to detect hallucinations and related observ- able overgeneration mistakes,

    M. Siino, “Brainllama at semeval-2024 task 6: Prompt- ing llama to detect hallucinations and related observ- able overgeneration mistakes,” inProceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024), 2024, pp. 82–87

    2024

  30. [31]

    Gpt hallucination detection through prompt engineering,

    M. Siino and I. Tinnirello, “Gpt hallucination detection through prompt engineering,” inProc. of the 25th Work- ing Notes of the Conference and Labs of the Evaluation Forum, vol. 3740, 2024, pp. 712–721

    2024

  31. [32]

    Factselfcheck: Fact-level black- box hallucination detection for llms,

    A. Sawczyn, J. Binkowski, D. Janiak, B. Gabrys, and T. Kajdanowicz, “Factselfcheck: Fact-level black- box hallucination detection for llms,”arXiv preprint arXiv:2503.17229, 2025

    arXiv 2025

  32. [33]

    Zero-knowledge llm hallucination detection and mitigation through fine-grained cross-model consistency,

    A. Goel, D. Schwartz, and Y. Qi, “Zero-knowledge llm hallucination detection and mitigation through fine-grained cross-model consistency,”arXiv preprint arXiv:2508.14314, 2025

    arXiv 2025

  33. [34]

    Counterfactual probing for hallucination de- tection and mitigation in large language models,

    Y. Feng, “Counterfactual probing for hallucination de- tection and mitigation in large language models,”arXiv preprint arXiv:2508.01862, 2025

    Pith/arXiv arXiv 2025

  34. [35]

    Large language models in health care: Development, applications, and challenges,

    R. Yang, T. F. Tan, W. Lu, A. J. Thirunavukarasu, D. S. W. Ting, and N. Liu, “Large language models in health care: Development, applications, and challenges,” Health Care Science, vol. 2, no. 4, pp. 255–263, 2023

    2023

  35. [36]

    Fact-and-reflection (far) improves confidence calibration of large language models,

    X. Zhao, H. Zhang, X. Pan, W. Yao, D. Yu, T. Wu, and J. Chen, “Fact-and-reflection (far) improves confidence calibration of large language models,”arXiv preprint arXiv:2402.17124, 2024

    Pith/arXiv arXiv 2024

  36. [37]

    Ethical and regulatory challenges of large language models in medicine,

    J. C. L. Ong, S. Y.-H. Chang, W. William, A. J. Butte, N. H. Shah, L. S. T. Chew, N. Liu, F. Doshi-Velez, W. Lu, J. Savulescuet al., “Ethical and regulatory challenges of large language models in medicine,”The Lancet Digital Health, vol. 6, no. 6, pp. e428–e432, 2024

    2024

  37. [38]

    A data-centric approach to generate faithful and high quality patient sum- maries with large language models,

    S. Hegselmann, S. Z. Shen, F. Gierse, M. Agrawal, D. Sontag, and X. Jiang, “A data-centric approach to generate faithful and high quality patient sum- maries with large language models,”arXiv preprint arXiv:2402.15422, 2024

    Pith/arXiv arXiv 2024

  38. [39]

    Deploy llama 3 8b with vllm,

    Red Hat Developer, “Deploy llama 3 8b with vllm,” Red Hat Developer, Jun. 2024, [Online]. Avail- able: https://developers.redhat.com/articles/2024/06/ 18/deploy-llama-3-8b-with-vllm

    2024

  39. [40]

    Benchmarking llm inference back- ends,

    DSS Solutions, “Benchmarking llm inference back- ends,” DSS Solutions Tech Blog, Jun. 2024, [On- line]. Available: https://dsssolutions.com/2024/06/17/ benchmarking-llm-inference-backends/

    2024

  40. [41]

    Evidencemap: a three-level knowledge representation for medical evidence computation and comprehension,

    T. Kang, Y. Sun, J. H. Kim, C. Ta, A. Perotte, K. Schif- fer, M. Wu, Y. Zhao, N. Moustafa-Fahmy, Y. Penget al., “Evidencemap: a three-level knowledge representation for medical evidence computation and comprehension,” Journal of the American Medical Informatics Associa- tion, vol. 30, no. 6, pp. 1022–1031, 2023

    2023

  41. [42]

    Automated fact check- ing: Task formulations, methods and future directions,

    J. Thorne and A. Vlachos, “Automated fact check- ing: Task formulations, methods and future directions,” arXiv preprint arXiv:1806.07687, 2018

    Pith/arXiv arXiv 2018

  42. [43]

    The perils and promises of fact-checking with large language models,

    D. Quelle and A. Bovet, “The perils and promises of fact-checking with large language models,”Frontiers in Artificial Intelligence, vol. 7, p. 1341697, 2024

    2024

  43. [44]

    Scientific claim verification with fine- tuned nli models,

    M. Koˇ sprdi´ c, A. Ljaji´ c, D. Medvecki, B. Baˇ saragin, and N. Milosevic, “Scientific claim verification with fine- tuned nli models,” inProceedings of the 16th Interna- tional Conference on Agents and Artificial Intelligence (ICAART). SCITEPRESS, November 2024

    2024

  44. [45]

    Large language models in medicine,

    A. J. Thirunavukarasu, D. S. J. Ting, K. Elangovan, L. Gutierrez, T. F. Tan, and D. S. W. Ting, “Large language models in medicine,”Nature medicine, vol. 29, no. 8, pp. 1930–1940, 2023

    1930

  45. [46]

    A comprehensive survey of hal- lucination mitigation techniques in large language mod- els,

    S. Tonmoy, S. Zaman, V. Jain, A. Rani, V. Rawte, A. Chadha, and A. Das, “A comprehensive survey of hal- lucination mitigation techniques in large language mod- els,”arXiv preprint arXiv:2401.01313, January 2024

    Pith/arXiv arXiv 2024

  46. [47]

    Generative large language models in automated fact- checking: A survey,

    I. Vykopal, M. Pikuliak, S. Ostermann, and M. ˇSimko, “Generative large language models in automated fact- checking: A survey,”arXiv preprint arXiv:2407.02351, July 2024

    Pith/arXiv arXiv 2024

  47. [48]

    A survey on hallucination in large language models: Prin- ciples, taxonomy, challenges, and open questions,

    L. Huang, W. Yu, W. Ma, W. Zhong, Z. Feng, H. Wang, Q. Chen, W. Peng, X. Feng, B. Qin, and T. Liu, “A survey on hallucination in large language models: Prin- ciples, taxonomy, challenges, and open questions,”arXiv preprint arXiv:10.1145/3703155, 2023

    doi:10.1145/3703155 2023

  48. [49]

    Overview of the mediqa-corr 2024 shared task on medical error detection and correction,

    A. Ben Abacha, W.-W. Yim, Y. Fu, Z. Sun, F. Xia, 18 and M. Yetisgen, “Overview of the mediqa-corr 2024 shared task on medical error detection and correction,” inProceedings of the 2024 Conference on Clinical Natu- ral Language Processing (ClinicalNLP). Association for Computational Linguistics, January 2024, pp. 596–603

    2024

  49. [50]

    Bridg- ing pre-trained models to continual learning: A hyper- network based framework with parameter-efficient fine- tuning techniques,

    F. Ding, C. Xu, H. Liu, B. Zhou, and H. Zhou, “Bridg- ing pre-trained models to continual learning: A hyper- network based framework with parameter-efficient fine- tuning techniques,”Information Sciences, vol. 674, p. 120710, 2024

    2024

  50. [51]

    Bioportal: ontologies and in- tegrated data resources at the click of a mouse,

    N. F. Noy, N. H. Shah, P. L. Whetzel, B. Dai, M. Dorf, N. Griffith, C. Jonquet, D. L. Rubin, M.-A. Storey, C. G. Chute, and M. A. Musen, “Bioportal: ontologies and in- tegrated data resources at the click of a mouse,”Nucleic Acids Research, vol. 37, no. suppl 2, pp. W170–W173, 2009

    2009

  51. [52]

    Improving natural language arguments’ identification by leveraging semantic similarity,

    F. M. Schmidt, A. Cohen, S. Gottifredi, and A. J. Garc ´ ıa, “Improving natural language arguments’ identification by leveraging semantic similarity,”Information Sciences, p. 122954, 2025

    2025

  52. [53]

    Negation scope detection for sentiment analysis: A re- inforcement learning framework for replicating human interpretations,

    N. Pr¨ ollochs, S. Feuerriegel, B. Lutz, and D. Neumann, “Negation scope detection for sentiment analysis: A re- inforcement learning framework for replicating human interpretations,”Information Sciences, vol. 536, pp. 205– 221, 2020

    2020

  53. [54]

    Mimic-iii, a freely accessible critical care database,

    A. E. W. Johnson, T. J. Pollard, L. H. Shen, L. H. Lehman, M. Feng, M. Ghassemi, B. Moody, P. Szolovits, L. A. Celi, and R. G. Mark, “Mimic-iii, a freely accessible critical care database,”Scientific Data, vol. 3, p. 160035, 2016

    2016

  54. [55]

    Releasing claude instant 1.2,

    Anthropic, “Releasing claude instant 1.2,” https://www. anthropic.com/index/releasing-claude-instant-1-2, 2023, accessed: October 19, 2025

    2023

  55. [56]

    Claude 2,

    ——, “Claude 2,” https://www.anthropic.com/index/ claude-2, 2023, accessed: October 19, 2025

    2023

  56. [57]

    Code llama: Open foundation models for code,

    B. Roziere, J. Gehring, F. Gloeckle, S. Sootla, I. Gat, X. E. Tan, Y. Adi, J. Liu, R. Sauvestre, T. Remezet al., “Code llama: Open foundation models for code,”arXiv preprint arXiv:2308.12950, 2023

    Pith/arXiv arXiv 2023

  57. [58]

    Medalpaca – an open-source collection of medical conversational ai models and training data,

    T. Han, L. C. Adams, J.-M. Papaioannou, P. Grundmann, T. Oberhauser, A. Figueroa, A. L¨ oser, D. Truhn, and K. K. Bressem, “Medalpaca – an open-source collection of medical conversational ai models and training data,” 2025. [Online]. Available: https://arxiv.org/abs/2304.08247

    Pith/arXiv arXiv 2025

  58. [59]

    Clinical camel: An open expert-level medical language model with dialogue- based knowledge encoding,

    A. Toma, P. R. Lawler, J. Ba, R. G. Krishnan, B. B. Rubin, and B. Wang, “Clinical camel: An open expert-level medical language model with dialogue- based knowledge encoding,” 2023. [Online]. Available: https://arxiv.org/abs/2305.12031

    Pith/arXiv arXiv 2023

  59. [60]

    Ehrsql: A practical text-to-sql benchmark for electronic health records,

    G. Lee, H. Hwang, S. Bae, Y. Kwon, W. Shin, S. Yang, M. Seo, J.-Y. Kim, and E. Choi, “Ehrsql: A practical text-to-sql benchmark for electronic health records,”Ad- vances in Neural Information Processing Systems, vol. 35, pp. 15 589–15 601, 2022

    2022

  60. [61]

    Publicly shareable clinical large language model built on synthetic clinical notes,

    S. Kweon, J. Kim, J. Kim, S. Im, E. Cho, S. Bae, J. Oh, G. Lee, J. H. Moon, S. C. You, S. Baek, C. H. Han, Y. B. Jung, Y. Jo, and E. Choi, “Publicly shareable clinical large language model built on synthetic clinical notes,” 2024. [Online]. Available: https://arxiv.org/abs/2309.00237

    Pith/arXiv arXiv 2024

  61. [62]

    Llama 2: Open foundation and fine- tuned chat models,

    H. Touvron, L. Martin, K. Stone, P. Albert, A. Alma- hairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosaleet al., “Llama 2: Open foundation and fine- tuned chat models,”arXiv preprint arXiv:2307.09288, 2023. 19

    Pith/arXiv arXiv 2023

  62. [252]

    [Online]

    PMLR, 16–17 Aug 2024. [Online]. Available: https://proceedings.mlr.press/v252/zhang24a.html

    2024

  63. [2021]

    3784–3803

    Association for Computational Linguistics, 2021, pp. 3784–3803. [Online]. Available: https: //aclanthology.org/2021.findings-emnlp.320/

    2021