Pith. sign in

REVIEW 4 major objections 4 minor 50 references

MIRIAD: Augmenting LLMs with millions of medical query-response pairs

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Turning medical papers into millions of question-answer pairs makes retrieved knowledge more usable by LLMs and improves medical answers.

desk verdict A large medical QA dataset with a useful but not fully isolated RAG claim; the hallucination and retriever-training results need stronger controls before the format effect is proven. read the letter →

arxiv 2506.06091 v2 pith:OTS7TYWT submitted 2025-06-06 cs.CL

Qinyue Zheng , Salman Abdullah , Sam Rawal , Cyril Zakka , Sophie Ostmeier , Maximilian Purk , Eduardo Reis , Eric J. Topol
show 2 more authors
Jure Leskovec Michael Moor
This is my paper
classification cs.CL
keywords medicalquestionansweringretrieval-augmentedgenerationknowledgebaseinstruction-responsedatasethallucinationdetectionLLMgrounding
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces MIRIAD, a corpus of 5.8 million medical question-answer pairs generated from 812,384 peer-reviewed medical papers. The authors argue that converting unstructured medical text into a question-answer format makes medical knowledge more usable for retrieval-augmented LLMs. In their experiments, adding MIRIAD to a basic RAG pipeline improved multiple-choice medical QA accuracy by up to 6.7% relative to retrieving the same source corpus as raw passages, with the same number of retrieved tokens. It also improved LLMs' ability to detect medical hallucinations, raising F1 by 22.5 to 37 points. If the claim holds, the format of retrieved knowledge, not just its content, is a significant lever in medical AI.

What carries the argument

The central object is the operationalized QA pair: a question of roughly 15 to 20 words and an answer of roughly 60 to 80 words, generated from a passage of up to 1,000 tokens and linked to the source paper's metadata. The mechanism that carries the argument is the multi-stage quality-control pipeline: rule-based filtering removes questions that merely reference the passage, a classifier trained on a small set of language-model-annotated examples discards low-quality pairs, and human medical experts validate a subset. This pipeline is what turns raw literature into retrievable knowledge units, which the paper claims align better with the instruction-driven distribution of downstream medical QA tasks.

What would settle it

Re-run the RAG comparison using only QA pairs whose answers are word-by-word contained in their source passages, with no out-of-passage n-grams; if structured retrieval no longer beats raw-passage retrieval on that subset, the central format claim is falsified.

Watch

Extended reading notes

Core claim

MIRIAD's central claim is that operationalizing medical literature as query-response pairs, each grounded in one passage, improves retrieval-augmented generation over the same literature in unstructured chunks. The paper reports that RAG with MIRIAD beats RAG with raw passages across six combinations of embedding and generation models, with relative gains from 1.16% to 6.74% on a multiple-choice medical benchmark, and that the same corpus improves hallucination-detection F1 by 22.53 points on the full test set and 37.02 points on a human-annotated subset. The paper also reports that training a general-domain retriever on MIRIAD improves held-out retrieval metrics, and that the corpus can be visually explored through an interactive atlas organized by 56 medical disciplines.

Load-bearing premise

The load-bearing premise is that the millions of generated answers stay strictly inside their source passages; if the generating model smuggles in outside medical knowledge, the reported format advantage could actually be an advantage of extra knowledge rather than of structured retrieval.

Editorial extensions

If this is right

  • Smaller open-source LLMs can narrow the accuracy gap with much larger proprietary models when given structured MIRIAD knowledge in a fixed context budget.
  • MIRIAD can serve as a large-scale supervised dataset for training medical retrievers, improving held-out retrieval metrics over a general-domain baseline.
  • Medical hallucination detection can be improved by roughly 22 to 37 F1 points through retrieval alone, without fine-tuning the detector.
  • The structured QA format reduces harmful retrieval outcomes by 14.3% compared with raw-passage retrieval, and the benefit generalizes to two additional medical QA benchmarks beyond the main one.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the format itself is the active ingredient, then restructuring other unstructured domain corpora into query-response pairs could yield similar retrieval gains in fields like law, finance, or scientific literature.
  • The low overlap of beneficial retrieved samples across model configurations suggests the gain may come from the overall QA format and context density rather than from any specific retrieved pair, a distinction the paper leaves open.
  • A direct test of the grounding claim would be to compare RAG performance on QA pairs whose answers are strictly contained in their source passages against QA pairs that introduce outside medical information; if only the latter help, the advantage is knowledge injection, not format.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces MIRIAD, a corpus of 5.82 million medical question-answer pairs automatically generated with GPT-3.5-Turbo from passages of medical articles in S2ORC, followed by rule-based filtering, LLM-based quality classification, and a small human validation study. The authors report that using MIRIAD as a retrieval corpus in a minimalist RAG pipeline improves accuracy on MedMCQA by up to 6.7% relative to retrieving unstructured passages from the same source corpus, improves hallucination-detection F1 by 22.5–37 points on MedHallu, and supports retriever training and an interactive atlas. The central claim is that encoding medical knowledge in an operationalized QA format, rather than as unstructured text, is what drives the retrieval and reasoning gains.

Significance. The contribution is a large, open, literature-grounded medical QA corpus at a scale unmatched by existing datasets, and the paper provides a reproducible RAG evaluation with fixed token budgets, two embedders, three backbone LLMs, and confidence intervals across multiple benchmarks. If the format-based causal claim is correct, MIRIAD would be a useful resource for medical RAG, retriever training, and hallucination detection. However, the current evidence does not fully isolate the QA format from the fact that MIRIAD answers are generated by GPT-3.5 and may contain knowledge beyond the source passage, so the headline comparison supports the corpus's utility but not yet the mechanistic conclusion the paper draws.

major comments (4)
  1. [Section 2.3, Table 1; Section 4.2] The RAG comparison does not isolate retrieval format from retrieved content. RAG-MIRIAD retrieves GPT-3.5-Turbo-generated answers, while RAG-Passage retrieves the original literature chunks; these differ not only in structure but in content, because generated answers are not guaranteed to be faithful to the source passage. The quality-control evidence is too weak to rule out substantial ungrounded content: only 168 QA pairs were human-validated, unanimous inter-annotator agreement was 46.7% for factuality and 60% for groundedness, and the Mistral-7B classifier has 81.8% recall and 69.7% precision. At the 5.8M scale, even a small ungroundedness rate yields hundreds of thousands of answers that inject model knowledge beyond the passage. Therefore the observed +1.16% to +6.74% relative gains could reflect extra content rather than the operationalized format. To support the central claim, the authors should provide a controlled comparison that holds content fixed (e.g., retrieve MIRIAD QA pairs with the answer text removed or replaced by the passage text; compare MIRIAD-5.8M vs MIRIAD-4.4M; or measure answer-passage lexical/semantic overlap and show gains persist on a strictly grounded subset).
  2. [Section 2.5, Table 2] The hallucination-detection experiment lacks a RAG-Passage condition. The comparison is LLM+MIRIAD vs LLM alone, so the +22.53 and +37.02 F1 improvements cannot be attributed to the structured QA format; they may simply reflect that adding any relevant retrieved evidence helps the model verify factual content. Adding a RAG-Passage arm with the same 3 retrieved items and the same token budget is necessary before claiming that MIRIAD's structure, rather than the presence of retrieved evidence, improves hallucination detection.
  3. [Section 2.4, Fig. 5; Section 4.4 (Retriever training)] The retriever-training proof-of-concept is evaluated on a held-out split of MIRIAD's own synthetic QA distribution. This demonstrates that the model can fit and retrieve within the same distribution it was trained on, but it does not show that MIRIAD improves retrieval on external medical queries. To support the claim that MIRIAD can be used to train medical retrievers, the authors should evaluate the trained retriever on an external benchmark (for example, retrieval recall for MedQA/MedMCQA questions against the source corpus, or a BEIR-style medical retrieval task).
  4. [Section 2.1 vs Section 4.2 (Human annotation)] The human-annotation results are reported inconsistently. Section 2.1 and Fig. 2d report human-LLM agreement of 92.3% on groundedness, 88.6% on factuality, and 78.4% on relevance, while Section 4.2 reports 81.3% on groundedness, 76.0% on factuality, and 93.3% on relevance for the same overlapping set. These numbers should be reconciled. In addition, the low unanimous inter-annotator agreement (46.7% factuality, 60% groundedness) should be acknowledged in the main text rather than only in Methods, because it materially affects the strength of the corpus-quality claim.
minor comments (4)
  1. [Section 2.3] The text reads 'leveraging MIRAD QA pairs' where the dataset name should be MIRIAD; the same typo appears in Supplementary Figures S7–S18.
  2. [Section 4.2] The precision definition is confusing: the sentence '69.7% of the examples were correctly identified as low quality by the classifier' repeats the recall definition and should be rewritten as '69.7% of the examples flagged as low quality by the classifier were indeed GPT-4-labeled bad examples.'
  3. [Section 4.1] The authors state they used the 'first 894,352 articles' from the filtered S2ORC pool; since the order is not described, it is unclear whether this is effectively a random sample or a potentially biased prefix, and this should be clarified.
  4. [Table 2] The table header layout is visually ambiguous: 'LLM+MIRIAD F1 Accuracy LLM Alone (∆F1)' mixes headers and values; please split into separate columns for F1 and Accuracy under each condition.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; central RAG and hallucination results are externally benchmarked.

full rationale

MIRIAD's principal claims—improved RAG accuracy and hallucination detection—are evaluated against external benchmarks (MedMCQA, MedQA-USMLE, MMLU-Med, MedHallu), so they are not self-referential. The RAG comparison holds the source corpus and token budget fixed, although the QA-generation step means the retrieved text is not literally identical in content; this is a confounding factor, not a circular reduction. The quality-control pipeline is validated on a small human-annotated sample, providing external (if limited) evidence. The retriever-training proof-of-concept in Fig. 5 evaluates on a held-out split of the same synthetic distribution, which is self-referential in scope, but the paper explicitly labels it a proof-of-concept and does not use it to support the central RAG claims. No load-bearing self-citations, uniqueness theorems, or definitional equivalences were found.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central claims rest on assumptions about the reliability of LLM-generated QA pairs, the adequacy of the filtering pipeline, and the validity of the evaluation benchmarks. No numerical parameters are fitted to the benchmark results, but the QA generation prompt, filtering thresholds, and model choices are external inputs.

assumptions (4)
  • domain assumption The S2ORC 'Medicine' category filter isolates relevant medical literature.
    The dataset is built solely from papers tagged as Medicine in S2ORC; if the tag is noisy or incomplete, the coverage and grounding of the QA pairs are affected. Invoked in Section 4.1.
  • domain assumption GPT-4-0613 labels are a valid proxy for factual correctness, relevance, and groundedness.
    The Mistral-7B classifier is trained on GPT-4 labels, and human agreement is only measured on 168 QA pairs with unanimous agreement as low as 46.7% for factuality. This is the basis of the quality filter in Section 4.2.
  • domain assumption GPT-3.5-Turbo-generated answers are grounded in the source passage and factually correct.
    The QA generation prompt instructs grounding, but without per-pair verification, ungrounded or hallucinated content may enter the dataset, especially in the 5.8M version that skips the classifier filter. See Sections 4.1 and 4.2.
  • domain assumption MedMCQA, MMLU-Med, MedQA-USMLE, and MedHallu are valid benchmarks for medical QA and hallucination detection.
    The evaluations rely on these external benchmarks to estimate the utility of MIRIAD; if the benchmarks are not representative of clinical use, the conclusions may not transfer. Used in Sections 2.3 to 2.5.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MIRIAD: Augmenting LLMs with millions of medical query-response pairs." pith.science (2026). https://pith.science/paper/OTS7TYWT

@misc{pith2026250606091,
  author       = {Pith},
  title        = {Pith review of: MIRIAD: Augmenting LLMs with millions of medical query-response pairs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OTS7TYWT}},
  note         = {Machine review of arXiv:2506.06091}
}
read the original abstract

LLMs are bound to transform healthcare with advanced decision support and flexible chat assistants. However, LLMs are prone to generate inaccurate medical content. To ground LLMs in high-quality medical knowledge, LLMs have been equipped with external knowledge via RAG, where unstructured medical knowledge is split into small text chunks that can be selectively retrieved and integrated into the LLMs context. Yet, existing RAG pipelines rely on raw, unstructured medical text, which can be noisy, uncurated and difficult for LLMs to effectively leverage. Systematic approaches to organize medical knowledge to best surface it to LLMs are generally lacking. To address these challenges, we introduce MIRIAD, a large-scale, curated corpus of 5,821,948 medical QA pairs, each rephrased from and grounded in a passage from peer-reviewed medical literature using a semi-automated pipeline combining LLM generation, filtering, grounding, and human annotation. Unlike prior medical corpora, which rely on unstructured text, MIRIAD encapsulates web-scale medical knowledge in an operationalized query-response format, which enables more targeted retrieval. Experiments on challenging medical QA benchmarks show that augmenting LLMs with MIRIAD improves accuracy up to 6.7% compared to unstructured RAG baselines with the same source corpus and with the same amount of retrieved text. Moreover, MIRIAD improved the ability of LLMs to detect medical hallucinations by 22.5 to 37% (increase in F1 score). We further introduce MIRIAD-Atlas, an interactive map of MIRIAD spanning 56 medical disciplines, enabling clinical users to visually explore, search, and refine medical knowledge. MIRIAD promises to unlock a wealth of down-stream applications, including medical information retrievers, enhanced RAG applications, and knowledge-grounded chat interfaces, which ultimately enables more reliable LLM applications in healthcare.

Figures

Figures reproduced from arXiv: 2506.06091 by the authors.

Figure 1
Figure 1. Overview of the data generation pipeline of MIRIAD. From top to bottom the MIRIAD processing pipeline is shown. For dataset generation, 894,352 full-text articles from the medical literature were prepared for LLM processing by partitioning the articles into passage chunks of up to 1,000 text tokens. From each passage, an LLM generated medically-related question-answer (QA) pairs, grounded in the passage. In a second… view at source ↗
Figure 2
Figure 2. MIRIAD data characteristics. a, Temporal distribution of discipline-specific question–answer (QA) pairs in the MIRIAD dataset. The dataset spans 56 distinct medical domains, which have been grouped into 10 broader categories for clarity in this visualization. This figure illustrates the relationship between the medical domain of each QA pair and the publication year of the corresponding source paper. The top panel s… view at source ↗
Figure 3
Figure 3. Overview of the MIRIAD atlas. a, MIRIAD-Atlas is an interactive user interface for visually exploring MIRIAD. Each data point represents a question-answer (QA) pair that is grounded in an article from the literature, and is linked to the source passage and metadata (such as the venue, or the date of publication). To facilitate visualization, a subset of 1M data points are displayed. Each data point was classified in… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Topic-level analysis of RAG performance on MedMCQA. a, RAG-MIRIAD. b, RAG-Passage. To assess the effect of information structure in retrieval-augmented generation (RAG), we compare RAG-MIRIAD (top panel), which uses concise QA pairs from MIRIAD, with RAG-Passage (botto…
Figure 5
Figure 5. Figure 5: Demonstration of retriever training use case. MIRIAD can be leveraged as a large-scale supervised retrieval dataset to train improved medical information retrieval models. To our knowledge, this has previously not been possible, as large-scale medical retrieval data is…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

50 extracted references · 32 canonical work pages

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report

    work page

  2. [2]

    Ms marco: A human generated machine reading comprehension dataset

    Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, et al. Ms marco: A human generated machine reading comprehension dataset. 2016. Preprint athttps://arxiv.org/abs/1611.09268

    arXiv 2016

  3. [3]

    Large language models in machine translation

    Thorsten Brants, Ashok Popat, Peng Xu, Franz Josef Och, and Jeffrey Dean. Large language models in machine translation. InProceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL), pages 858– 867, 2007

    work page 2007

  4. [4]

    Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901, 2020

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901, 2020. 14

    work page 1901

  5. [5]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. InProceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pages 4171–4186, 2019

    2019

  6. [6]

    The llama 3 herd of models

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. 2024. Preprint athttps://arxiv.org/abs/2407.21783

    arXiv 2024

  7. [7]

    Jina-colbert-v2: A general-purpose multilingual late interaction retriever

    Rohan Jha, Bo Wang, Michael Günther, Georgios Mastrapas, Saba Sturua, Isabelle Mohr, Andreas Koukounas, Mohammad Kalim Akram, Nan Wang, and Han Xiao. Jina-colbert-v2: A general-purpose multilingual late interaction retriever. InThe 4th Workshop on Multilingual Representation Learning, page 159, 2024

    work page 2024

  8. [8]

    What disease does this patient have? a large-scale open domain question answering dataset from medical exams

    Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. Applied Sciences, 11(14):6421, 2021

    2021

Show all 50 references
  1. [9]

    Pubmedqa: A dataset for biomedical research question answering

    Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. Pubmedqa: A dataset for biomedical research question answering. InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natura...

    2019

  2. [10]

    Colbert: Efficient and effective passage search via contextualized late interaction over bert

    Omar Khattab and Matei Zaharia. Colbert: Efficient and effective passage search via contextualized late interaction over bert. InProceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, pages 39–48, 2020

    2020

  3. [11]

    Medexqa: Medical question answering benchmark with multiple explanations

    Yunsoo Kim, Jinge Wu, Yusuf Abdulle, and Honghan Wu. Medexqa: Medical question answering benchmark with multiple explanations. 2024. Preprint at https://arxiv.org/abs/2406. 06331

    2024

  4. [12]

    Bioasq-qa: A manually curated corpus for biomedical question answering.Scientific Data, 10(1):170, 2023

    Anastasia Krithara, Anastasios Nentidis, Konstantinos Bougiatiotis, and Georgios Paliouras. Bioasq-qa: A manually curated corpus for biomedical question answering.Scientific Data, 10(1):170, 2023

    2023

  5. [13]

    Halueval: A large-scale hallucination evaluation benchmark for large language models

    Junyi Li, Xiaoxue Cheng, Wayne Xin Zhao, Jian-Yun Nie, and Ji-Rong Wen. Halueval: A large-scale hallucination evaluation benchmark for large language models. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6449–6464, 2023

    2023

  6. [14]

    Retrieval augmented generation or long-context llms? a comprehensive study and hybrid approach

    Zhuowan Li, Cheng Li, Mingyang Zhang, Qiaozhu Mei, and Michael Bendersky. Retrieval augmented generation or long-context llms? a comprehensive study and hybrid approach. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track, p...

    2024

  7. [15]

    Can large language models reason about medical questions?Patterns, 5(3), 2024

    Valentin Liévin, Christoffer Egeberg Hother, Andreas Geert Motzfeldt, and Ole Winther. Can large language models reason about medical questions?Patterns, 5(3), 2024. 15

    2024

  8. [16]

    Deepseek-v3 technical report

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. 2024. Preprint at https: //arxiv.org/abs/2412.19437

    2024 arXiv

  9. [17]

    Query rewriting via large language models

    Jie Liu and Barzan Mozafari. Query rewriting via large language models. 2024. Preprint at https: //arxiv.org/abs/2403.09060

    2024

  10. [18]

    S2orc: The semantic scholar open research corpus

    Kyle Lo, Lucy Lu Wang, Mark Neumann, Rodney Kinney, and Daniel S Weld. S2orc: The semantic scholar open research corpus. InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4969–4983, 2020

    2020

  11. [19]

    Rafe: ranking feedback improves query rewriting for rag

    Shengyu Mao, Yong Jiang, Boli Chen, Xiao Li, Peng Wang, Xinyu Wang, Pengjun Xie, Fei Huang, Huajun Chen, and Ningyu Zhang. Rafe: ranking feedback improves query rewriting for rag. 2024. Preprint athttps://arxiv.org/abs/2405.14431

    2024 arXiv

  12. [20]

    Med-flamingo: a multimodal medical few-shot learner

    Michael Moor, Qian Huang, Shirley Wu, Michihiro Yasunaga, Yash Dalmia, Jure Leskovec, Cyril Zakka, Eduardo Pontes Reis, and Pranav Rajpurkar. Med-flamingo: a multimodal medical few-shot learner. InMachine Learning for Health (ML4H), pages 353–367. PMLR, 2023

    2023

  13. [21]

    Hybrid retrieval-augmented generation approach for llms query response enhancement

    Pouria Omrani, Alireza Hosseini, Kiana Hooshanfar, Zahra Ebrahimian, Ramin Toosi, and Moham- mad Ali Akhaee. Hybrid retrieval-augmented generation approach for llms query response enhancement. In2024 10th International Conference on Web Research (ICWR), pages 22–26. IEEE, 2024

    2024

  14. [22]

    Openbiollms: Advancing open-source large lan- guage models for healthcare and life sciences

    Ankit Pal and Malaikannan Sankarasubbu. Openbiollms: Advancing open-source large lan- guage models for healthcare and life sciences. https://huggingface.co/aaditya/ Llama3-OpenBioLLM-70B, 2024

    2024

  15. [23]

    Medmcqa: A large-scale multi- subject multi-choice dataset for medical domain question answering

    Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. Medmcqa: A large-scale multi- subject multi-choice dataset for medical domain question answering. InConference on health, inference, and learning, pages 248–260. PMLR, 2022

    2022

  16. [24]

    Med-halt: Medical domain hallucination test for large language models

    Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. Med-halt: Medical domain hallucination test for large language models. 2023. Preprint at https://arxiv.org/abs/2307. 15343

    2023

  17. [25]

    Medhallu: A comprehensive benchmark for detecting medical hallucinations in large language models

    Shrey Pandit, Jiawei Xu, Junyuan Hong, Zhangyang Wang, Tianlong Chen, Kaidi Xu, and Ying Ding. Medhallu: A comprehensive benchmark for detecting medical hallucinations in large language models

  18. [26]

    The fineweb datasets: Decanting the web for the finest text data at scale

    Guilherme Penedo, Hynek Kydlí ˇcek, Anton Lozhkov, Margaret Mitchell, Colin A Raffel, Leandro V on Werra, Thomas Wolf, et al. The fineweb datasets: Decanting the web for the finest text data at scale. Advances in Neural Information Processing Systems, 37:30811–30849, 2024

    2024

  19. [27]

    Qdrant - vector database.https://qdrant.tech/

    QdrantTeam. Qdrant - vector database.https://qdrant.tech/. Accessed: 2025-06-04

    2025

  20. [28]

    Language models are unsupervised multitask learners.OpenAI blog, 1(8):9, 2019

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners.OpenAI blog, 1(8):9, 2019. 16

    2019

  21. [29]

    Colbertv2: Effective and efficient retrieval via lightweight late interaction

    Keshav Santhanam, Omar Khattab, Jon Saad-Falcon, Christopher Potts, and Matei Zaharia. Colbertv2: Effective and efficient retrieval via lightweight late interaction. InProceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguis...

    2022

  22. [30]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. 2023. Preprint athttps://arxiv.org/abs/2302.13971

    2023 arXiv

  23. [31]

    Text embeddings by weakly-supervised contrastive pre-training

    Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. Text embeddings by weakly-supervised contrastive pre-training. 2022. Preprint at https://arxiv.org/abs/2212.03533

    2022 arXiv

  24. [32]

    Minilm: Deep self- attention distillation for task-agnostic compression of pre-trained transformers.Advances in neural information processing systems, 33:5776–5788, 2020

    Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. Minilm: Deep self- attention distillation for task-agnostic compression of pre-trained transformers.Advances in neural information processing systems, 33:5776–5788, 2020

    2020

  25. [33]

    Pmc-llama: to- ward building open-source language models for medicine.Journal of the American Medical Informatics Association, 31(9):1833–1843, 2024

    Chaoyi Wu, Weixiong Lin, Xiaoman Zhang, Ya Zhang, Weidi Xie, and Yanfeng Wang. Pmc-llama: to- ward building open-source language models for medicine.Journal of the American Medical Informatics Association, 31(9):1833–1843, 2024

    2024

  26. [34]

    C-pack: Packed resources for general chinese embeddings

    Shitao Xiao, Zheng Liu, Peitian Zhang, Niklas Muennighoff, Defu Lian, and Jian-Yun Nie. C-pack: Packed resources for general chinese embeddings. InProceedings of the 47th international ACM SIGIR conference on research and development in information retrieval, pages 641–649, 2024

    2024

  27. [35]

    Benchmarking retrieval-augmented generation for medicine

    Guangzhi Xiong, Qiao Jin, Zhiyong Lu, and Aidong Zhang. Benchmarking retrieval-augmented generation for medicine. InFindings of the Association for Computational Linguistics ACL 2024, pages 6233–6251, 2024

    2024

  28. [36]

    Medicine

    Cyril Zakka, Rohan Shad, Akash Chaurasia, Alex R Dalal, Jennifer L Kim, Michael Moor, Robyn Fong, Curran Phillips, Kevin Alexander, Euan Ashley, et al. Almanac—retrieval-augmented language models for clinical medicine.Nejm ai, 1(2):AIoa2300068, 2024. 4 Methods 4.1 Dataset gene...

    2024

  29. [39]

    Factual Accuracy: The answers should be factually correct and accurate

  30. [40]

    Streamlit

    Relevance: The Q&A should refer to medically relevant content. Classifier training based on LLM annotation We finetuned Mistral-7B-Instruct-v0.2 on the GPT-4 labels and were able to filter the 5,821,948 question answer pairs further down to 4,487,542. To validate the classifie...

  31. [41]

    bge-base-en-v1.5

    ( denoted as BGE-Large), which was specifically trained for retrieval tasks. Text embeddings were indexed within a Qdrant vector database [ 27] using cosine similarity to facilitate retrieval of the relevant entries. To ensure a fair comparison between different RAG corpus, we...

    2024

  32. [42]

    28 Prompt for Quality Control - Relevance Check (Cont’d)

    Refers to the details of a specific study (that is presented in the passage), such as specific details about the study’s experimental design, the used statistical methods used in this study, tables or figures that appear in it, study dates, locations, funding sources, or other...

  33. [43]

    Is heavily dependent on study-specific details that cannot be understood without the original passage, such as discussing the study’s specific findings, limitations, or conclusions without providing sufficient background information

  34. [44]

    good" or

    Focuses on experimental methods or protocols that, while medically related, are too specific to the referenced study and do not convey broadly relevant medical knowledge. If the Q&A does not have any other bad aspects mentioned above, then the Q&A should be classified as good....

  35. [45]

    Provide a short explanation for your classification

    incorrect Task: Using the criteria above, judge whether the Q&A is correct or incorrect. Provide a short explanation for your classification. S2.4 Groundedness prompt This prompt asks the model to determine whether the answer in a question-answer is grounded in the provided pa...

  36. [46]

    Every piece of information in the answer can be traced back to the provided passage

  37. [47]

    The answer does not include any information that goes beyond the content of the passage A Q&A is considered ungrounded if:

  38. [48]

    The answer contains any information not present in the passage

  39. [49]

    The answer makes claims or assertions that cannot be directly supported by the passage

  40. [50]

    The” or “In

    The answer extrapolates beyond what is stated in the passage Task: Judge whether the Q&A is grounded or ungrounded based on the passage provided. Provide a short explanation for your classification. Figure S1:Human annotation results. For three evaluation metrics, we report th...

  41. [2023]

    Preprint athttps://arxiv.org/abs/2303.08774

    arXiv

  42. [2025]

    Preprint athttps://arxiv.org/abs/2502.14302

    arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.