REVIEW 3 major objections 6 minor 27 references
DC3 -- A Diagnostic Case Challenge Collection for Clinical Decision Support
T0 review · 3 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read DC3 gives researchers 31 hard diagnostic cases with confirmed diagnoses and dense relevance labels across PubMed.
desk verdict A genuinely useful new case collection with confirmed diagnoses, but its dense relevance labels are built on an unvalidated mention-to-relevance rule, so the benchmark numbers should be read with caution until label precision is demonstrated. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the DC3 collection itself: 31 cases, each a temporally ordered sequence of physician-authored episode paragraphs ending in a confirmed diagnosis expressed as UMLS CUIs. The mechanism that makes dense relevance feasible is the inference rule that a PubMed document is relevant to a case if its title or abstract mentions any of the case's target diagnoses, applied at scale by a proprietary named entity recognition system. Because this rule is applied to the full 2018 PubMed snapshot of 27 million abstracts, it produces complete (dense) relevance labels with no pooling or manual assessment. The collection is distributed as a Python script that downloads the public NEJM case content and organizes it as JSON, with relevance judgments in standard trec_eval format.
What would settle it
Take a random sample of documents the NER labeled non-relevant for a given DC3 case and have clinicians judge them for diagnostic usefulness; if a substantial fraction actually describe the confirmed diagnosis using synonyms or indirect terminology, or if a manual audit shows the NER misses many true mentions, the dense relevance judgments are not a reliable ground truth for benchmarking.
Extended reading notes
Core claim
DC3 is a corpus of 31 difficult diagnostic case challenges collected at Massachusetts General Hospital between 2013 and 2018, originally published in NEJM's Case Challenge section. Each case is split into topically coherent paragraphs ('episodes') reflecting the temporal order of findings, hypotheses, and tests, and is labeled with the confirmed true diagnosis encoded as Unified Medical Language System concept unique identifiers (UMLS CUIs), averaging 6.9 target CUIs per case. The distinctive contribution is the dense relevance labeling: using a proprietary medical named entity recognition system, the authors marked every PubMed title or abstract that mentions any target diagnosis as relevant to that case, yielding a median of 3,597 relevant documents per case and up to 219,540 for the most multi-diagnosis outlier. They argue this contrasts with existing collections (TREC CDS, CLEF eHealth, MIMIC-III, i2b2), which rely on small or single-narrative cases, lack confirmed target diagnoses, or lack dense judgments, and therefore no comparable public resource existed. The paper frames DC3 as a benchmark for two tasks: patient-centric literature retrieval and supervised diagnosis classification, with baseline results demonstrating the difficulty of both.
Load-bearing premise
The whole relevance labeling depends on the assumption that a paper whose title or abstract mentions one of the case's confirmed diagnoses is genuinely useful to the diagnosing physician, and that the proprietary medical named entity recognizer finds those mentions without meaningful error rates.
Editorial extensions
If this is right
- Researchers can benchmark diagnostic decision support systems on authentic, difficult cases with fully specified relevance judgments, without needing private hospital data.
- The dense labels make it possible to train and evaluate retrieval systems that optimize directly for bringing the correct diagnosis to the physician's attention, rather than approximating relevance through sparse pooled judgments.
- Because the cases are rare or multi-diagnosis, the collection targets exactly the situations where diagnostic errors are most common and most harmful.
- The baseline results (best retrieval nDCG ~0.42, SVM classification F1 0.17 at k=500) suggest substantial headroom, so the collection can support future comparisons as methods improve.
Reading between the lines
- If the relevance-by-diagnosis-mention rule is accepted, the same NER pipeline could be rerun against future PubMed snapshots, turning DC3 into a living benchmark that tracks new literature as it appears — a step the paper does not itself take.
- The collection's emphasis on rare and imported infectious diseases may also make it useful for differential-diagnosis research in low-resource settings, where those conditions are more common, although the paper frames it only as a U.S. tertiary-care challenge.
- A direct extension would be to compare DC3's inferred labels against manual judgments on a subset, which would either validate the dense-labeling shortcut or quantify how much noise it introduces for downstream evaluation.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper presents DC3, a collection of 31 diagnostic case challenges taken from the New England Journal of Medicine's Case Challenge section, all drawn from Massachusetts General Hospital between 2013 and 2018. Each case consists of temporally ordered, physician-authored paragraphs (presenting complaint, investigations, etc.), the confirmed final diagnosis represented as UMLS concept unique identifiers, and inferred binary relevance labels covering the full 2018 PubMed snapshot of roughly 27 million abstracts. The labels are generated by a proprietary medical named entity recognition system under the rule that any paper whose title or abstract mentions a target CUI is relevant. The paper reports baseline results for patient-centric literature retrieval (TF-IDF, LambdaMART, DRMM) in terms of nDCG and for a diagnostic classification task (Naive Bayes, Logistic Regression, SVM) in terms of F1, and argues that no comparable publicly available collection existed at the time of writing, because existing resources (TREC CDS, CLEF eHealth, MIMIC-III, i2b2) lack dense relevance judgments and confirmed diagnoses for difficult, temporally structured cases.
Significance. If the relevance labels were validated, DC3 would fill a genuine gap: it is the only openly available collection offering authentic, difficult diagnostic episodes with confirmed diagnoses and dense relevance judgments over a realistically sized document collection, and its distribution format (JSON case files plus trec_eval relevance files) makes it easy to use. Strengths of the paper are its transparency about the inference-based nature of the labels (Section 5(1)), its explicit per-case documentation in Table 1, and its choice of real, published cases with confirmed diagnoses. The baseline experiments are honestly described as illustrative rather than competitive. The central risk is that the dense labels, which are the unique selling point, rest on an unevaluated proprietary system and a mention-equals-relevance assumption, so the benchmark numbers in Tables 2 and 3 are not yet interpretable as measuring diagnostic decision support.
major comments (3)
- [Section 3.1, footnote 1] The relevance labels that support the collection's central benchmarking claim are produced by a proprietary NER system whose precision and recall are never reported; footnote 1 asserts only "generally high performance" without numbers, and no manual verification or inter-annotator agreement is provided. Since Table 2's nDCG values and Table 3's training labels are computed against these judgments, the reported baseline numbers inherit whatever error rate the system has; the authors' own concession in Section 5(1) that "purely inferred relevance judgments do not replace manual expert annotations" confirms that this is a known limitation rather than a resolved one. The paper should either evaluate the NER system on a held-out sample, or provide a manually validated sample of relevance judgments (for instance, a few hundred documents per case stratified by CUI) and report agreement statistics, before the dense labels can be used as a benchmark.
- [Section 3.1, Table 1 (Case 20)] The mention-equals-relevance rule conflates "a paper mentions a diagnosis" with "a paper is useful for this patient's diagnostic challenge." Target CUI lists include incidental and background conditions: Case 20 has 29 CUIs including coronary arteriosclerosis, papillary renal-cell carcinoma, and old sarcoidosis, yielding 219,540 "relevant" documents, so a system could score well by retrieving generic literature on the most common comorbidity rather than on the actual diagnostic problem (granulomatous amebic encephalitis). Negative or differential mentions (e.g., "ruled out Wilson's disease" in Case 6) would also be counted as relevant by the current rule. The authors should report label precision on a stratified sample, and should consider restricting the relevance definition to the primary differential or weighting the target CUIs, or at minimum discuss how much the composition of the CUI lists affects the nDCG scores in Table 2.
- [Tables 2 and 3] The reported baseline scores are not sufficiently specified to be interpreted. Table 2 reports only a single aggregate nDCG value per model across 31 cases that vary enormously in label density (135 to 219,540 relevant documents; median 3,597), with no cutoff parameter, no per-case breakdown, and no significance or variance information; aggregate nDCG over such a skewed collection could be dominated by a few outlier cases. Table 3 omits the train/test split, the document representation, and whether F1 is macro- or micro-averaged, and it is unclear whether the case notes themselves appear among the PubMed training abstracts (the NEJM case challenges are indexed in PubMed). These details matter because the classification and retrieval results are presented as evidence of task difficulty and as a reference for future benchmarking.
minor comments (6)
- [Section 3.2, Table 1] The collection name is spelled inconsistently as "DC3" in most places but "DC 3" in the Section 3.2 heading and caption of Figure 1; one spelling should be used throughout.
- [References] Reference [13] has a garbled author list ("Engineering National Academies of Sciences, Medicine, et al."); it should be formatted as the National Academies of Sciences, Engineering, and Medicine.
- [Abstract and Section 3.1] The abstract says the cases were "manually compiled and solved by clinical experts," but Section 3.1 states that the authors did not perform manual sub-selection and included all published MGH case challenges; the provenance (that the cases come from NEJM's Case Challenge authors) should be clarified to avoid implying that the paper's authors authored the case narratives.
- [Figure 1] Figure 1 is referenced but the example JSON is not visible in the arXiv version of the paper, so the claimed distribution format cannot be inspected from the manuscript itself.
- [Section 3.2] The release materials (the Python script and the inferred relevance judgments) are not accompanied by a stated license, and only NEJM's copyright on the underlying case content is mentioned; the authors should specify the usage terms for the labels and code.
- [Section 4.1] The paper does not specify the nDCG cutoff (e.g., nDCG@10 or nDCG@1000), and with dense binary labels the reported score is sensitive to this choice, so the evaluation setup should be stated.
Circularity Check
No significant circularity; DC3's construction, labels, and baselines are self-contained, with only external validity caveats about inferred relevance labels.
full rationale
The paper builds a diagnostic case collection from externally confirmed MGH/NEJM case diagnoses, represents them as UMLS CUIs, and uses a proprietary NER system to label PubMed documents as relevant whenever the title or abstract mentions those CUIs. This is a stated annotation rule, not a fitted parameter or a prediction derived from the labels. The retrieval and classification baselines are measured against those labels, but they do not use the labels as training input in any circular way: Lucene indexing, TF-IDF, LambdaMART, DRMM, Naive Bayes, Logistic Regression, and SVM are standard methods evaluated on the explicitly defined relevance criterion. The paper's claim that such inferred dense judgments enable benchmarking is an external-validity assertion, not a circular derivation, and the authors explicitly concede in Section 5 that 'Purely inferred relevance judgments do not replace manual expert annotations.' The main scientific concern is label precision—whether a mention of a target CUI is always genuinely relevant, especially with incidental co-morbidities—but that is a measurement-validity issue, not circular reasoning. No self-citation is load-bearing for the dataset construction, and no uniqueness theorem is invoked to force a choice. The central contribution is a resource, and its derivation chain is self-contained apart from the unevaluated NER, which is a quality risk rather than a circularity.
Assumptions & free parameters
assumptions (4)
- domain assumption Any PubMed paper whose title or abstract mentions a target diagnosis is relevant to the case.
- domain assumption The proprietary NER system identifies all mentions of target diagnosis CUIs with generally high performance.
- domain assumption The NEJM Case Challenge articles provide authentic, temporally ordered physician observations reflecting actual diagnostic episodes at MGH.
- domain assumption Target diagnoses listed by the original case authors are correct and complete for benchmarking.
Cite this review
Pith. "Pith review of DC3 -- A Diagnostic Case Challenge Collection for Clinical Decision Support." pith.science (2026). https://pith.science/paper/IIOKD2HC
@misc{pith2026190808581,
author = {Pith},
title = {Pith review of: DC3 -- A Diagnostic Case Challenge Collection for Clinical Decision Support},
year = {2026},
howpublished = {\url{https://pith.science/paper/IIOKD2HC}},
note = {Machine review of arXiv:1908.08581}
}
read the original abstract
In clinical care, obtaining a correct diagnosis is the first step towards successful treatment and, ultimately, recovery. Depending on the complexity of the case, the diagnostic phase can be lengthy and ridden with errors and delays. Such errors have a high likelihood to cause patients severe harm or even lead to their death and are estimated to cost the U.S. healthcare system several hundred billion dollars each year. To avoid diagnostic errors, physicians increasingly rely on diagnostic decision support systems drawing from heuristics, historic cases, textbooks, clinical guidelines and scholarly biomedical literature. The evaluation of such systems, however, is often conducted in an ad-hoc fashion, using non-transparent methodology, and proprietary data. This paper presents DC3, a collection of 31 extremely difficult diagnostic case challenges, manually compiled and solved by clinical experts. For each case, we present a number of temporally ordered physician-generated observations alongside the eventually confirmed true diagnosis. We additionally provide inferred dense relevance judgments for these cases among the PubMed collection of 27 million scholarly biomedical articles.
Figures
Reference graph
Works this paper leans on
-
[1]
Brown university at trec precision medicine 2018
Prakrit Baruah, Riya Dulepet, Kyle Qian, and Carsten Eickhoff. Brown university at trec precision medicine 2018. InProceedings of the 27th Text Retrieval Conference (TREC), 2018
work page 2018
-
[2]
Biomedical question answering via weighted neural network passage retrieval
Ferenc Galkó and Carsten Eickhoff. Biomedical question answering via weighted neural network passage retrieval. In Proceedings of the 40th European Conference on Information Retrieval (ECIR) . Springer, 2018
work page 2018
-
[3]
Tejal K Gandhi, Allen Kachalia, Eric J Thomas, Ann Louise Puopolo, Catherine Yoon, Troyen A Brennan, and David M Studdert. Missed and delayed diagnoses in the ambulatory setting: a study of closed malpractice claims.Annals of internal medicine, 145(7):488–496, 2006
work page 2006
-
[4]
Neural document embeddings for intensive care patient mortality prediction
Paulina Grnarova, Florian Schmidt, Stephanie L Hyland, and Carsten Eickhoff. Neural document embeddings for intensive care patient mortality prediction. NIPS Workshop on Machine Learning for Health , 2016
work page 2016
-
[5]
Cumulated gain-based evaluation of ir techniques
Kalervo Järvelin and Jaana Kekäläinen. Cumulated gain-based evaluation of ir techniques. ACM TOIS, 20(4):422–446, 2002
work page 2002
-
[6]
Mimic-iii, a freely accessible critical care database
Alistair EW Johnson, Tom J Pollard, Lu Shen, H Lehman Li-wei, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. Mimic-iii, a freely accessible critical care database. Scientific data, 3:160035, 2016
work page 2016
-
[7]
Diagnostic overshadowing reviewed and reconsidered
David A Jopp and Christopher B Keys. Diagnostic overshadowing reviewed and reconsidered. American Journal on Mental Retardation , 106(5):416–433, 2001
work page 2001
-
[8]
Diagnostic difficulty and error in primary care?a systematic review
Olga Kostopoulou, Brendan C Delaney, and Craig W Munro. Diagnostic difficulty and error in primary care?a systematic review. Family practice, 25(6), 2008
work page 2008
Show all 27 references
-
[9]
Implicit negative feedback in clinical infor- mation retrieval
Lorenz Kuhn and Carsten Eickhoff. Implicit negative feedback in clinical infor- mation retrieval. In Proceedings of the ACM SIGIR Medical Information Retrieval Workshop, 2016
2016
-
[10]
Research in ambulatory patient safety 2000–2010: a 10-year review
CY Lorincz, E Drazen, PE Sokol, KV Neerukonda, J Metzger, MC Toepp, L Maul, DC Classen, and MK Wynia. Research in ambulatory patient safety 2000–2010: a 10-year review. Chicago, IL: American Medical Association , 2011
2000
-
[11]
Best care at lower cost: the path to continuously learning health care in America
J Michael McGinnis, Leigh Stuckhardt, Robert Saunders, Mark Smith, et al. Best care at lower cost: the path to continuously learning health care in America. National Academies Press, 2013
2013
-
[12]
Machine learning for real-time prediction of complications in critical care: a retrospective study
Alexander Meyer, Dina Zverinski, Boris Pfahringer, Jörg Kempfert, Titus Kuehne, Simon H Sündermann, Christof Stamm, Thomas Hofmann, Volkmar Falk, and Carsten Eickhoff. Machine learning for real-time prediction of complications in critical care: a retrospective study. The Lance...
2018
-
[13]
National Academies Press, 2016
Engineering National Academies of Sciences, Medicine, et al.Improving diagnosis in health care. National Academies Press, 2016
2016
-
[14]
Diagnostic errors?the next frontier for patient safety
David E Newman-Toker and Peter J Pronovost. Diagnostic errors?the next frontier for patient safety. Jama, 301(10):1060–1062, 2009
2009
-
[15]
Learning from malpractice claims about negligent, adverse events in primary care in the united states
Robert L Phillips, Lori A Bartholomew, Susan M Dovey, GE Fryer, TJ Miyoshi, and LA Green. Learning from malpractice claims about negligent, adverse events in primary care in the united states. BMJ Quality & Safety , 13(2):121–126, 2004
2004
-
[16]
The frequency and nature of medical error in primary care: understanding the diversity across studies
John Sandars and Aneez Esmail. The frequency and nature of medical error in primary care: understanding the diversity across studies. Family practice, 2003
2003
-
[17]
Challenges of making a diagnosis in the outpatient setting: a multi-site survey of primary care physicians
Urmimala Sarkar, Doug Bonacum, William Strull, Christiane Spitzmueller, Nancy Jin, Andrea López, Traber Davis Giardina, Ashley ND Meyer, and Hardeep Singh. Challenges of making a diagnosis in the outpatient setting: a multi-site survey of primary care physicians. BMJ Qual Saf,...
2011
-
[18]
What went wrong in 1,000 negligence claims
N Silk. What went wrong in 1,000 negligence claims. Health care risk report , 7:13–15, 2000
2000
-
[19]
Overview of the trec 2014 clinical decision support track
Matthew S Simpson, Ellen M Voorhees, and William Hersh. Overview of the trec 2014 clinical decision support track. Technical report, DTIC Document, 2014
2014
-
[20]
Types and origins of diagnostic errors in primary care settings
Hardeep Singh, Traber Davis Giardina, Ashley ND Meyer, Samuel N Forjuoh, Michael D Reis, and Eric J Thomas. Types and origins of diagnostic errors in primary care settings. JAMA internal medicine, 173(6):418–425, 2013
2013
-
[21]
Reducing diagnostic error through medical home-based primary care reform
Hardeep Singh and Mark Graber. Reducing diagnostic error through medical home-based primary care reform. Jama, 304(4):463–464, 2010
2010
-
[22]
Identifying risk factors for heart disease over time: Overview of 2014 i2b2/uthealth shared task track 2
Amber Stubbs, Christopher Kotfila, Hua Xu, and Özlem Uzuner. Identifying risk factors for heart disease over time: Overview of 2014 i2b2/uthealth shared task track 2. Journal of biomedical informatics , 58:S67–S77, 2015
2014
-
[23]
Overview of the share/clef ehealth evaluation lab 2013
Hanna Suominen, Sanna Salanterä, Sumithra Velupillai, Wendy W Chapman, Guergana Savova, Noemie Elhadad, Sameer Pradhan, Brett R South, Danielle L Mowery, Gareth JF Jones, et al. Overview of the share/clef ehealth evaluation lab 2013. In International Conference of the Cross-La...
2013
-
[24]
Recognizing obesity and comorbidities in sparse data
Özlem Uzuner. Recognizing obesity and comorbidities in sparse data. Journal of the American Medical Informatics Association , 16(4):561–570, 2009
2009
-
[25]
Identifying patient smoking status from medical discharge records
Özlem Uzuner, Ira Goldstein, Yuan Luo, and Isaac Kohane. Identifying patient smoking status from medical discharge records. Journal of the American Medical Informatics Association, 15(1):14–24, 2008
2008
-
[26]
Embedding electronic health records for clinical information retrieval
Xing Wei and Carsten Eickhoff. Embedding electronic health records for clinical information retrieval. arXiv preprint arXiv:1811.05402, 2018
2018 arXiv
-
[27]
Improving ambulatory patient safety: learning from the last decade, moving ahead in the next
Matthew K Wynia and David C Classen. Improving ambulatory patient safety: learning from the last decade, moving ahead in the next. Jama, 306(22), 2011
2011
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.