{"id":"adb20d57-e060-43d5-bd35-08eeeffadb3f","arxiv_id":"2506.06091","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A 5.8M-pair, literature-grounded medical QA corpus improves RAG accuracy and hallucination detection in LLMs compared with unstructured passage retrieval.","lead":"Researchers generated 5.8 million medical question-and-answer pairs from over 800,000 peer-reviewed papers and tested them as a retrieval source for medical AI. When fed to large language models, the QA pairs improved multiple-choice medical accuracy by up to 6.7% over raw-text retrieval and boosted hallucination-detection F1 scores by 22 to 37 points.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"RAG gains may reflect GPT-3.5 knowledge injection rather than the QA format; the same-source comparison does not isolate structure because QA generation changes content.","rationale":"The reader's weakest assumption identifies the same load-bearing threat: the RAG comparisons and the hallucination experiment may be explained by GPT-3.5 injecting non-passage knowledge into MIRIAD's answers, rather than by the operationalized QA format. This is the right concern because the paper's own quality-control numbers in Section 4.2 show that groundedness and factuality are not perfectly enforced at scale: the classifier has 81.8% recall and 69.7% precision on a 15,000-sample GPT-4 label set, and the human validation set of 168 QA pairs has low unanimous agreement on factuality (46.7%) and groundedness (60%). The paper also states in the Discussion that semi-synthetic generation 'inevitably includes some non-reducible inaccuracies,' confirming that content contamination is acknowledged in scope. The hallucination experiment lacks any RAG-Passage baseline, so the large F1 gains in Table 2 are especially vulnerable to the confound. I do not think this requires rejecting the paper's headline accuracy result: the consistent advantage across six configurations and additional benchmarks in Section S3.5 is real evidence, and the proposed control experiment could confirm the format explanation. Therefore the reader's CONDITIONAL verdict should stand, with the concrete test above as a natural condition for strengthening acceptance.","tokens_in":28862,"tokens_out":4918,"duration_ms":51618,"concrete_test":"Re-run the MedMCQA RAG comparison with a controlled version of MIRIAD in which every answer is replaced by an extractive summary of the source passage (or all GPT-3.5 answers are filtered by an entailment/groundedness system that discards any answer sentence not supported by the passage), using identical embedders, context budget, and prompts. If the RAG-MIRIAD advantage over RAG-Passage largely disappears, the gain is due to injected knowledge rather than format. In addition, add a RAG-Passage arm to the MedHallu experiment; without it, the hallucination F1 improvement is uninterpretable as a format effect.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that representing medical text as QA pairs, rather than unstructured passages, drives RAG gains. The comparison in Section 2.3 and Table 1 is meant to hold the source corpus fixed, but it does not hold the actual retrieved text fixed: RAG-MIRIAD retrieves GPT-3.5-Turbo-generated answers, which are not guaranteed to be faithful to the source passage. The quality-control pipeline (Section 4.2) trains a Mistral-7B classifier on only 15,000 GPT-4 labels, with 81.8% recall and 69.7% precision, and human validation covers only 168 QA pairs, with unanimous agreement as low as 46.7% for factuality and 60% for groundedness. At the 5.8M scale, even a modest ungroundedness rate yields hundreds of thousands of QA answers containing information beyond the passage. If retrieved MIRIAD answers inject medically relevant world knowledge, the observed 1.16-6.74% relative accuracy gains over RAG-Passage could reflect additional content, not the operationalized format. The hallucination experiment (Table 2) is particularly exposed: it compares LLM+MIRIAD against LLM alone, with no RAG-Passage condition, so the +22.5 to +37 F1 gains cannot be attributed to structure; they may simply reflect that any retrieved relevant evidence helps. Thus the strongest claim's causal interpretation rests on an unverified equivalence between 'rephrased from' and 'containing only passage content.'","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces MIRIAD, a corpus of 5.82 million medical question-answer pairs automatically generated with GPT-3.5-Turbo from passages of medical articles in S2ORC, followed by rule-based filtering, LLM-based quality classification, and a small human validation study. The authors report that using MIRIAD as a retrieval corpus in a minimalist RAG pipeline improves accuracy on MedMCQA by up to 6.7% relative to retrieving unstructured passages from the same source corpus, improves hallucination-detection F1 by 22.5–37 points on MedHallu, and supports retriever training and an interactive atlas. The central claim is that encoding medical knowledge in an operationalized QA format, rather than as unstructured text, is what drives the retrieval and reasoning gains.","tokens_in":29110,"tokens_out":4190,"duration_ms":40333,"significance":"The contribution is a large, open, literature-grounded medical QA corpus at a scale unmatched by existing datasets, and the paper provides a reproducible RAG evaluation with fixed token budgets, two embedders, three backbone LLMs, and confidence intervals across multiple benchmarks. If the format-based causal claim is correct, MIRIAD would be a useful resource for medical RAG, retriever training, and hallucination detection. However, the current evidence does not fully isolate the QA format from the fact that MIRIAD answers are generated by GPT-3.5 and may contain knowledge beyond the source passage, so the headline comparison supports the corpus's utility but not yet the mechanistic conclusion the paper draws.","major_comments":[{"comment":"The RAG comparison does not isolate retrieval format from retrieved content. RAG-MIRIAD retrieves GPT-3.5-Turbo-generated answers, while RAG-Passage retrieves the original literature chunks; these differ not only in structure but in content, because generated answers are not guaranteed to be faithful to the source passage. The quality-control evidence is too weak to rule out substantial ungrounded content: only 168 QA pairs were human-validated, unanimous inter-annotator agreement was 46.7% for factuality and 60% for groundedness, and the Mistral-7B classifier has 81.8% recall and 69.7% precision. At the 5.8M scale, even a small ungroundedness rate yields hundreds of thousands of answers that inject model knowledge beyond the passage. Therefore the observed +1.16% to +6.74% relative gains could reflect extra content rather than the operationalized format. To support the central claim, the authors should provide a controlled comparison that holds content fixed (e.g., retrieve MIRIAD QA pairs with the answer text removed or replaced by the passage text; compare MIRIAD-5.8M vs MIRIAD-4.4M; or measure answer-passage lexical/semantic overlap and show gains persist on a strictly grounded subset).","section":"Section 2.3, Table 1; Section 4.2"},{"comment":"The hallucination-detection experiment lacks a RAG-Passage condition. The comparison is LLM+MIRIAD vs LLM alone, so the +22.53 and +37.02 F1 improvements cannot be attributed to the structured QA format; they may simply reflect that adding any relevant retrieved evidence helps the model verify factual content. Adding a RAG-Passage arm with the same 3 retrieved items and the same token budget is necessary before claiming that MIRIAD's structure, rather than the presence of retrieved evidence, improves hallucination detection.","section":"Section 2.5, Table 2"},{"comment":"The retriever-training proof-of-concept is evaluated on a held-out split of MIRIAD's own synthetic QA distribution. This demonstrates that the model can fit and retrieve within the same distribution it was trained on, but it does not show that MIRIAD improves retrieval on external medical queries. To support the claim that MIRIAD can be used to train medical retrievers, the authors should evaluate the trained retriever on an external benchmark (for example, retrieval recall for MedQA/MedMCQA questions against the source corpus, or a BEIR-style medical retrieval task).","section":"Section 2.4, Fig. 5; Section 4.4 (Retriever training)"},{"comment":"The human-annotation results are reported inconsistently. Section 2.1 and Fig. 2d report human-LLM agreement of 92.3% on groundedness, 88.6% on factuality, and 78.4% on relevance, while Section 4.2 reports 81.3% on groundedness, 76.0% on factuality, and 93.3% on relevance for the same overlapping set. These numbers should be reconciled. In addition, the low unanimous inter-annotator agreement (46.7% factuality, 60% groundedness) should be acknowledged in the main text rather than only in Methods, because it materially affects the strength of the corpus-quality claim.","section":"Section 2.1 vs Section 4.2 (Human annotation)"}],"minor_comments":[{"comment":"The text reads 'leveraging MIRAD QA pairs' where the dataset name should be MIRIAD; the same typo appears in Supplementary Figures S7–S18.","section":"Section 2.3"},{"comment":"The precision definition is confusing: the sentence '69.7% of the examples were correctly identified as low quality by the classifier' repeats the recall definition and should be rewritten as '69.7% of the examples flagged as low quality by the classifier were indeed GPT-4-labeled bad examples.'","section":"Section 4.2"},{"comment":"The authors state they used the 'first 894,352 articles' from the filtered S2ORC pool; since the order is not described, it is unclear whether this is effectively a random sample or a potentially biased prefix, and this should be clarified.","section":"Section 4.1"},{"comment":"The table header layout is visually ambiguous: 'LLM+MIRIAD F1 Accuracy LLM Alone (∆F1)' mixes headers and values; please split into separate columns for F1 and Accuracy under each condition.","section":"Table 2"}],"recommendation":"major_revision","confidential_remarks":"The resource itself is potentially valuable and the main RAG evaluation is internally consistent across multiple models and benchmarks. The key issue is causal attribution: the paper claims the QA format drives the gains, but the experimental design does not separate format from retrieved content. This is fixable with additional controlled experiments (e.g., passage-answer matching, 4.4M vs 5.8M comparison, and a RAG-Passage arm in the hallucination experiment), so I see major revision rather than rejection. The inconsistent human-agreement numbers in Sections 2.1 and 4.2 should be resolved before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: this is primarily a resource paper, and the resource looks genuinely useful; but the headline causal claim—that the QA format itself drives RAG gains—is not actually isolated, because the generated QA answers can contain more than the source passage. The reported accuracy gain is modest and consistent, the hallucination gain is large but missing the natural control.\n\nWhat's new: a 5.8M-pair medical QA dataset, each pair traced to a passage in a peer-reviewed paper. That scale with explicit provenance is new in the medical domain, and the pipeline (chunking, rule filtering, classifier filtering, human spot-check) is clearly described. The main RAG experiment is well executed as a descriptive comparison: fixed token budget, same source corpus, two embedders, three LLMs, three benchmarks—the structure advantage replicates, which is a real result. The dataset will likely be useful to the community, and the Atlas is a nice extra.\n\nSoft spots: (1) The central comparison does not hold the retrieved text fixed. RAG-MIRIAD retrieves GPT-3.5-Turbo-generated answers, which the paper's own validation cannot fully guarantee are grounded. Fifteen thousand GPT-4 labels, a Mistral-7B classifier at 81.8% recall / 69.7% precision, and 168 human-annotated pairs with unanimous agreement as low as 46.7% on factuality is thin armor for 5.8M pairs. Even a modest ungroundedness rate injects extra medical knowledge into the QA format, so the RAG gain could be a content effect, not a format effect. This is the main weakness and it's substantial. (2) The hallucination experiment compares LLM+MIRIAD against LLM alone; any retrieved relevant evidence might produce a similar F1 jump, so a RAG-Passage condition is needed before crediting the format. (3) The retriever-training demonstration shows learning curves on held-out QA pairs from the same synthetic distribution; that is a proof-of-concept, not evidence that MIRIAD-trained retrievers help on real queries or RAG. (4) The data is not yet downloadable, so at review time the central artifact is unverifiable.\n\nNone of these are fatal to the resource itself. They are fixable: release the data, add the missing baseline, strengthen the human validation with more pairs and higher agreement, and benchmark the trained retriever externally. The paper is honest about its limits and the main result is worth engaging with. I'd send it to a serious referee, expecting major revision; the dataset deserves to enter the literature with stronger caveats.","headline":"A large medical QA dataset with a useful but not fully isolated RAG claim; the hallucination and retriever-training results need stronger controls before the format effect is proven.","tokens_in":29720,"tokens_out":4017,"would_cite":true,"duration_ms":40459,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Turning medical papers into millions of question-answer pairs makes retrieved knowledge more usable by LLMs and improves medical answers.","keywords":["medical question answering","retrieval-augmented generation","medical knowledge base","instruction-response dataset","hallucination detection","LLM grounding"],"falsifier":"Re-run the RAG comparison using only QA pairs whose answers are word-by-word contained in their source passages, with no out-of-passage n-grams; if structured retrieval no longer beats raw-passage retrieval on that subset, the central format claim is falsified.","tokens_in":28636,"feed_emoji":"🩺","tokens_out":5258,"duration_ms":52506,"temperature":0.7,"pith_summary":"This paper introduces MIRIAD, a corpus of 5.8 million medical question-answer pairs generated from 812,384 peer-reviewed medical papers. The authors argue that converting unstructured medical text into a question-answer format makes medical knowledge more usable for retrieval-augmented LLMs. In their experiments, adding MIRIAD to a basic RAG pipeline improved multiple-choice medical QA accuracy by up to 6.7% relative to retrieving the same source corpus as raw passages, with the same number of retrieved tokens. It also improved LLMs' ability to detect medical hallucinations, raising F1 by 22.5 to 37 points. If the claim holds, the format of retrieved knowledge, not just its content, is a significant lever in medical AI.","feed_headline":"Medical QA pairs beat raw text in retrieval-augmented LLMs","feed_subtitle":"A 5.8M-pair literature-grounded corpus lifts RAG accuracy up to 6.7% and hallucination-detection F1 by 22–37 points.","key_machinery":"The central object is the operationalized QA pair: a question of roughly 15 to 20 words and an answer of roughly 60 to 80 words, generated from a passage of up to 1,000 tokens and linked to the source paper's metadata. The mechanism that carries the argument is the multi-stage quality-control pipeline: rule-based filtering removes questions that merely reference the passage, a classifier trained on a small set of language-model-annotated examples discards low-quality pairs, and human medical experts validate a subset. This pipeline is what turns raw literature into retrievable knowledge units, which the paper claims align better with the instruction-driven distribution of downstream medical QA tasks.","core_discovery":"MIRIAD's central claim is that operationalizing medical literature as query-response pairs, each grounded in one passage, improves retrieval-augmented generation over the same literature in unstructured chunks. The paper reports that RAG with MIRIAD beats RAG with raw passages across six combinations of embedding and generation models, with relative gains from 1.16% to 6.74% on a multiple-choice medical benchmark, and that the same corpus improves hallucination-detection F1 by 22.53 points on the full test set and 37.02 points on a human-annotated subset. The paper also reports that training a general-domain retriever on MIRIAD improves held-out retrieval metrics, and that the corpus can be visually explored through an interactive atlas organized by 56 medical disciplines.","pith_inferences":["If the format itself is the active ingredient, then restructuring other unstructured domain corpora into query-response pairs could yield similar retrieval gains in fields like law, finance, or scientific literature. ","The low overlap of beneficial retrieved samples across model configurations suggests the gain may come from the overall QA format and context density rather than from any specific retrieved pair, a distinction the paper leaves open. ","A direct test of the grounding claim would be to compare RAG performance on QA pairs whose answers are strictly contained in their source passages against QA pairs that introduce outside medical information; if only the latter help, the advantage is knowledge injection, not format. "],"forward_implications":["Smaller open-source LLMs can narrow the accuracy gap with much larger proprietary models when given structured MIRIAD knowledge in a fixed context budget. ","MIRIAD can serve as a large-scale supervised dataset for training medical retrievers, improving held-out retrieval metrics over a general-domain baseline. ","Medical hallucination detection can be improved by roughly 22 to 37 F1 points through retrieval alone, without fine-tuning the detector. ","The structured QA format reduces harmful retrieval outcomes by 14.3% compared with raw-passage retrieval, and the benefit generalizes to two additional medical QA benchmarks beyond the main one. "],"supporting_citations":[{"why":"Supplies the medical literature pool from which every QA pair is generated.","marker":"[18]"},{"why":"Provides the main multiple-choice benchmark used for the RAG comparison.","marker":"[23]"},{"why":"Provides the medical hallucination-detection benchmark used for the F1 comparisons.","marker":"[25]"},{"why":"Provides one of the embedding models used to encode queries and corpus entries.","marker":"[32]"},{"why":"Provides the retrieval-tuned embedding model used in the main RAG and hallucination experiments.","marker":"[34]"},{"why":"Powers the vector search used to retrieve QA pairs and passages in the RAG pipeline.","marker":"[27]"}],"fun_headline_variants":["5.8M grounded medical QA pairs boost RAG accuracy by 6.7%","Medical QA pairs outperform raw text in RAG, lifting F1 up to 37%","MIRIAD: 5.8M literature-grounded QA pairs sharpen medical RAG","QA-formatted corpus boosts RAG accuracy and hallucination detection"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the millions of generated answers stay strictly inside their source passages; if the generating model smuggles in outside medical knowledge, the reported format advantage could actually be an advantage of extra knowledge rather than of structured retrieval.","fun_headline_variants_meta":{"raw":{"variants":["5.8M grounded medical QA pairs boost RAG accuracy by 6.7%","Medical QA pairs outperform raw text in RAG, lifting F1 up to 37%","MIRIAD: 5.8M literature-grounded QA pairs sharpen medical RAG","QA-formatted corpus boosts RAG accuracy and hallucination detection"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001005,"raw_usage":{"total_tokens":4317,"prompt_tokens":1078,"completion_tokens":3239,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":694,"completion_tokens_details":{"reasoning_tokens":3149}},"tokens_in":694,"tokens_out":3239,"duration_ms":22015,"temperature":1.0,"reasoning_tokens":3149,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T06:00:07.141143+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the RAG comparison using only QA pairs whose answers are word-by-word contained in their source passages, with no out-of-passage n-grams; if structured retrieval no longer beats raw-passage retrieval on that subset, the central format claim is falsified.","supporting_citations":[{"cited_title":"S2orc: The semantic scholar open research corpus","cited_arxiv_id":null,"evidence_quote":"Supplies the medical literature pool from which every QA pair is generated."},{"cited_title":"Medmcqa: A large-scale multi- subject multi-choice dataset for medical domain question answering","cited_arxiv_id":null,"evidence_quote":"Provides the main multiple-choice benchmark used for the RAG comparison."},{"cited_title":"Medhallu: A comprehensive benchmark for detecting medical hallucinations in large language models","cited_arxiv_id":null,"evidence_quote":"Provides the medical hallucination-detection benchmark used for the F1 comparisons."},{"cited_title":"Qdrant - vector database.https://qdrant.tech/","cited_arxiv_id":null,"evidence_quote":"Powers the vector search used to retrieve QA pairs and passages in the RAG pipeline."}],"review_version":1}