{"id":"0c935b18-b6ef-4589-9c9e-cd5b7eea8222","arxiv_id":"1908.08581","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"DC3 introduces a public benchmark of 31 hard clinical cases with confirmed diagnoses and dense relevance judgments over the full PubMed collection.","lead":"DC3 is a collection of 31 deliberately difficult diagnostic case challenges from Massachusetts General Hospital, each with physician-written observations, a confirmed diagnosis, and dense relevance labels covering all 27 million PubMed abstracts. Researchers who build or test clinical decision support systems can use it as a public benchmark instead of private or small-scale evaluation data.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The dense relevance labels rest on an unevaluated proprietary NER and a mention-equals-relevance rule; without a manual validation sample, the benchmark's evaluation numbers are not trustworthy.","rationale":"The reader's weakest assumption correctly identifies the load-bearing premise: relevance is inferred from a proprietary NER's detection of target-diagnosis mentions, with no reported accuracy evaluation. My stress-test confirms this and sharpens it. The concern is not that the authors are dishonest; the paper is openly work-in-progress and includes a limitation statement. The issue is that the central claim—a usable public benchmark with dense relevance judgments—depends on label quality that is currently unverified. In particular, the mention-equals-relevance rule creates likely false positives from negated mentions and incidental co-morbidities, not just false negatives from NER misses. That directly affects the validity of the reported nDCG and F1 numbers. The proposed manual validation test would settle whether the concern lands. Since the reader's conditional verdict already requires NER evaluation and manual validation of a label sample, no verdict adjustment beyond that is needed; if the test fails badly, the appropriate outcome would be rejection or unverdictable status, but that determination is premature without the data.","tokens_in":6397,"tokens_out":3193,"duration_ms":38794,"concrete_test":"Draw a stratified random sample of, say, 500 judged-relevant and 500 judged-irrelevant PubMed IDs across the 31 cases (or 100 per case if feasible), with automated labels redacted. Have two clinicians independently judge whether each document would help a physician confirm or increase suspicion of the case's true presenting diagnosis. Compute inter-annotator agreement and the positive predictive value/recall of the automated labels. If PPV is materially below roughly 0.8, or if clinician agreement is low, the relevance labels and the Table 2 and Table 3 baselines should be re-reported after label correction. As a secondary check, re-run Table 2 with a public NER (e.g., SciSpacy) to see whether the nDCG ordering is robust to the label-generation method.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's distinctive contribution is the dense PubMed relevance judgments, which make the reported nDCG/F1 baselines possible and support the claim of 'robust and reproducible benchmarking' (Section 5). These labels are generated by a proprietary medical NER system under the rule that any paper whose title or abstract mentions a target CUI is relevant (Section 3.1). No NER performance figures are reported; the footnote only asserts 'generally high performance', and Section 5(1) concedes that inferred labels do not replace manual annotation. The load-bearing problem is not merely false negatives. A mention can occur in a negative or differential context (e.g., 'not Wilson's disease'), and case target lists include incidental co-morbidities—Case 20 lists coronary arteriosclerosis and papillary renal-cell carcinoma alongside amebic encephalitis, yielding 219,540 'relevant' documents. Since nDCG is computed against these labels, a system can score well by retrieving generic literature about incidental findings. Thus Table 2 does not currently measure diagnostic decision support as claimed until label precision is demonstrated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents DC3, a collection of 31 diagnostic case challenges taken from the New England Journal of Medicine's Case Challenge section, all drawn from Massachusetts General Hospital between 2013 and 2018. Each case consists of temporally ordered, physician-authored paragraphs (presenting complaint, investigations, etc.), the confirmed final diagnosis represented as UMLS concept unique identifiers, and inferred binary relevance labels covering the full 2018 PubMed snapshot of roughly 27 million abstracts. The labels are generated by a proprietary medical named entity recognition system under the rule that any paper whose title or abstract mentions a target CUI is relevant. The paper reports baseline results for patient-centric literature retrieval (TF-IDF, LambdaMART, DRMM) in terms of nDCG and for a diagnostic classification task (Naive Bayes, Logistic Regression, SVM) in terms of F1, and argues that no comparable publicly available collection existed at the time of writing, because existing resources (TREC CDS, CLEF eHealth, MIMIC-III, i2b2) lack dense relevance judgments and confirmed diagnoses for difficult, temporally structured cases.","tokens_in":6587,"tokens_out":8817,"duration_ms":79131,"significance":"If the relevance labels were validated, DC3 would fill a genuine gap: it is the only openly available collection offering authentic, difficult diagnostic episodes with confirmed diagnoses and dense relevance judgments over a realistically sized document collection, and its distribution format (JSON case files plus trec_eval relevance files) makes it easy to use. Strengths of the paper are its transparency about the inference-based nature of the labels (Section 5(1)), its explicit per-case documentation in Table 1, and its choice of real, published cases with confirmed diagnoses. The baseline experiments are honestly described as illustrative rather than competitive. The central risk is that the dense labels, which are the unique selling point, rest on an unevaluated proprietary system and a mention-equals-relevance assumption, so the benchmark numbers in Tables 2 and 3 are not yet interpretable as measuring diagnostic decision support.","major_comments":[{"comment":"The relevance labels that support the collection's central benchmarking claim are produced by a proprietary NER system whose precision and recall are never reported; footnote 1 asserts only \"generally high performance\" without numbers, and no manual verification or inter-annotator agreement is provided. Since Table 2's nDCG values and Table 3's training labels are computed against these judgments, the reported baseline numbers inherit whatever error rate the system has; the authors' own concession in Section 5(1) that \"purely inferred relevance judgments do not replace manual expert annotations\" confirms that this is a known limitation rather than a resolved one. The paper should either evaluate the NER system on a held-out sample, or provide a manually validated sample of relevance judgments (for instance, a few hundred documents per case stratified by CUI) and report agreement statistics, before the dense labels can be used as a benchmark.","section":"Section 3.1, footnote 1"},{"comment":"The mention-equals-relevance rule conflates \"a paper mentions a diagnosis\" with \"a paper is useful for this patient's diagnostic challenge.\" Target CUI lists include incidental and background conditions: Case 20 has 29 CUIs including coronary arteriosclerosis, papillary renal-cell carcinoma, and old sarcoidosis, yielding 219,540 \"relevant\" documents, so a system could score well by retrieving generic literature on the most common comorbidity rather than on the actual diagnostic problem (granulomatous amebic encephalitis). Negative or differential mentions (e.g., \"ruled out Wilson's disease\" in Case 6) would also be counted as relevant by the current rule. The authors should report label precision on a stratified sample, and should consider restricting the relevance definition to the primary differential or weighting the target CUIs, or at minimum discuss how much the composition of the CUI lists affects the nDCG scores in Table 2.","section":"Section 3.1, Table 1 (Case 20)"},{"comment":"The reported baseline scores are not sufficiently specified to be interpreted. Table 2 reports only a single aggregate nDCG value per model across 31 cases that vary enormously in label density (135 to 219,540 relevant documents; median 3,597), with no cutoff parameter, no per-case breakdown, and no significance or variance information; aggregate nDCG over such a skewed collection could be dominated by a few outlier cases. Table 3 omits the train/test split, the document representation, and whether F1 is macro- or micro-averaged, and it is unclear whether the case notes themselves appear among the PubMed training abstracts (the NEJM case challenges are indexed in PubMed). These details matter because the classification and retrieval results are presented as evidence of task difficulty and as a reference for future benchmarking.","section":"Tables 2 and 3"}],"minor_comments":[{"comment":"The collection name is spelled inconsistently as \"DC3\" in most places but \"DC 3\" in the Section 3.2 heading and caption of Figure 1; one spelling should be used throughout.","section":"Section 3.2, Table 1"},{"comment":"Reference [13] has a garbled author list (\"Engineering National Academies of Sciences, Medicine, et al.\"); it should be formatted as the National Academies of Sciences, Engineering, and Medicine.","section":"References"},{"comment":"The abstract says the cases were \"manually compiled and solved by clinical experts,\" but Section 3.1 states that the authors did not perform manual sub-selection and included all published MGH case challenges; the provenance (that the cases come from NEJM's Case Challenge authors) should be clarified to avoid implying that the paper's authors authored the case narratives.","section":"Abstract and Section 3.1"},{"comment":"Figure 1 is referenced but the example JSON is not visible in the arXiv version of the paper, so the claimed distribution format cannot be inspected from the manuscript itself.","section":"Figure 1"},{"comment":"The release materials (the Python script and the inferred relevance judgments) are not accompanied by a stated license, and only NEJM's copyright on the underlying case content is mentioned; the authors should specify the usage terms for the labels and code.","section":"Section 3.2"},{"comment":"The paper does not specify the nDCG cutoff (e.g., nDCG@10 or nDCG@1000), and with dense binary labels the reported score is sensitive to this choice, so the evaluation setup should be stated.","section":"Section 4.1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript's central evaluation resource depends on a proprietary NER system attributed to codiag AG, an organization with which two of the authors are affiliated; this commercial dependency is not disclosed beyond the affiliation line and deserves editorial attention because it limits the auditability and reproducibility of the main contribution. The reference list also contains a noticeable number of self-citations (references [1], [2], [4], [9], [12], and [26] are co-authored by one of the paper's authors), which is not inappropriate for a niche line of work but is worth a routine check. The stress-test concern about the unvalidated labels lands: the paper's own Section 5(1) concession, together with the sparse reporting in the footnote, makes the label-validation gap a genuine load-bearing issue for the benchmarking claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nDC3 is worth knowing about: it packages 31 real MGH diagnostic challenges (from NEJM Case Challenges) with confirmed final diagnoses and temporally ordered physician notes, and it provides a script that pulls the public cases into a JSON format. That alone is a useful contribution. The comparison to TREC CDS, CLEF eHealth, MIMIC-III, and i2b2 is fair, and the collection fills a real gap: no existing public set offers confirmed diagnoses on difficult, authentic primary-care-style cases.\n\nThe soft spot is the dense relevance judgments. The paper's headline feature is \"inferred dense relevance judgments\" for all 27M PubMed abstracts, generated by a proprietary medical NER under the rule that any paper whose title or abstract mentions a target CUI is relevant. That rule is never validated. The footnote only says the NER has \"generally high performance,\" with no precision/recall numbers, no manual sample check, no inter-annotator agreement. And the rule is blunt: a paper about \"ruling out Wilson's disease\" or about a co-morbidity incidental to the case (see Case 20, with coronary arteriosclerosis, papillary renal-cell carcinoma, etc., yielding 219,540 \"relevant\" documents) would count as relevant. Since nDCG in Table 2 is computed against these labels, a system can score well by retrieving generic literature about an incidental finding rather than literature that would help the physician reach the right diagnosis. So the evaluation numbers need to be read with real caution.\n\nThat said, the paper is honest. Section 5 explicitly says inferred labels do not replace manual expert annotations, and the authors frame this as work in progress. The dataset itself is still useful for training and for studying behavior on difficult cases, even if the labels are noisy. But for a benchmark that \"allows for robust and reproducible benchmarking,\" the inference chain needs to be exposed: evaluate the NER, sample and manually validate the labels, report per-case label precision, and give confidence intervals for the baselines. I'd also want the relevance judgments released with checksums and versioning.\n\nMy take: send it to peer review as a dataset paper, but treat the evaluation results as illustrative until the labels are validated. If you work in clinical IR, cite the case collection; don't yet cite the nDCG numbers as ground truth.","headline":"A genuinely useful new case collection with confirmed diagnoses, but its dense relevance labels are built on an unvalidated mention-to-relevance rule, so the benchmark numbers should be read with caution until label precision is demonstrated.","tokens_in":7088,"tokens_out":3003,"would_cite":true,"duration_ms":29579,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"DC3 gives researchers 31 hard diagnostic cases with confirmed diagnoses and dense relevance labels across PubMed.","keywords":["diagnostic decision support","clinical information retrieval","test collection","relevance judgments","PubMed","rare diseases","dataset","evaluation benchmark"],"falsifier":"Take a random sample of documents the NER labeled non-relevant for a given DC3 case and have clinicians judge them for diagnostic usefulness; if a substantial fraction actually describe the confirmed diagnosis using synonyms or indirect terminology, or if a manual audit shows the NER misses many true mentions, the dense relevance judgments are not a reliable ground truth for benchmarking.","tokens_in":6234,"feed_emoji":"🩺","tokens_out":8425,"duration_ms":67584,"temperature":0.7,"pith_summary":"This paper introduces DC3, a publicly available collection of 31 unusually difficult diagnostic cases drawn from Massachusetts General Hospital, each with physician-written observations in temporal order and a confirmed final diagnosis. The authors' main claim is that DC3 fills a real gap: at the time of writing, no comparable test collection offered challenging authentic diagnostic episodes together with confirmed diagnoses and dense relevance judgments over a realistically large document collection. To make that possible, they label every one of the 27 million PubMed abstracts as relevant or not to each case by checking whether the abstract mentions any of the case's confirmed diagnoses. The paper also reports baseline retrieval and classification scores to show the task is hard, with the best retrieval model reaching an nDCG around 0.42. A sympathetic reader would care because reproducible, transparent evaluation data of this kind is what lets diagnostic decision support systems be compared fairly and improved.","feed_headline":"31 hard diagnostic cases get dense relevance labels across PubMed","feed_subtitle":"Researchers get 31 confirmed-diagnosis cases with dense relevance labels over 27 million PubMed abstracts.","key_machinery":"The central object is the DC3 collection itself: 31 cases, each a temporally ordered sequence of physician-authored episode paragraphs ending in a confirmed diagnosis expressed as UMLS CUIs. The mechanism that makes dense relevance feasible is the inference rule that a PubMed document is relevant to a case if its title or abstract mentions any of the case's target diagnoses, applied at scale by a proprietary named entity recognition system. Because this rule is applied to the full 2018 PubMed snapshot of 27 million abstracts, it produces complete (dense) relevance labels with no pooling or manual assessment. The collection is distributed as a Python script that downloads the public NEJM case content and organizes it as JSON, with relevance judgments in standard trec_eval format.","core_discovery":"DC3 is a corpus of 31 difficult diagnostic case challenges collected at Massachusetts General Hospital between 2013 and 2018, originally published in NEJM's Case Challenge section. Each case is split into topically coherent paragraphs ('episodes') reflecting the temporal order of findings, hypotheses, and tests, and is labeled with the confirmed true diagnosis encoded as Unified Medical Language System concept unique identifiers (UMLS CUIs), averaging 6.9 target CUIs per case. The distinctive contribution is the dense relevance labeling: using a proprietary medical named entity recognition system, the authors marked every PubMed title or abstract that mentions any target diagnosis as relevant to that case, yielding a median of 3,597 relevant documents per case and up to 219,540 for the most multi-diagnosis outlier. They argue this contrasts with existing collections (TREC CDS, CLEF eHealth, MIMIC-III, i2b2), which rely on small or single-narrative cases, lack confirmed target diagnoses, or lack dense judgments, and therefore no comparable public resource existed. The paper frames DC3 as a benchmark for two tasks: patient-centric literature retrieval and supervised diagnosis classification, with baseline results demonstrating the difficulty of both.","pith_inferences":["If the relevance-by-diagnosis-mention rule is accepted, the same NER pipeline could be rerun against future PubMed snapshots, turning DC3 into a living benchmark that tracks new literature as it appears — a step the paper does not itself take.","The collection's emphasis on rare and imported infectious diseases may also make it useful for differential-diagnosis research in low-resource settings, where those conditions are more common, although the paper frames it only as a U.S. tertiary-care challenge.","A direct extension would be to compare DC3's inferred labels against manual judgments on a subset, which would either validate the dense-labeling shortcut or quantify how much noise it introduces for downstream evaluation."],"forward_implications":["Researchers can benchmark diagnostic decision support systems on authentic, difficult cases with fully specified relevance judgments, without needing private hospital data.","The dense labels make it possible to train and evaluate retrieval systems that optimize directly for bringing the correct diagnosis to the physician's attention, rather than approximating relevance through sparse pooled judgments.","Because the cases are rare or multi-diagnosis, the collection targets exactly the situations where diagnostic errors are most common and most harmful.","The baseline results (best retrieval nDCG ~0.42, SVM classification F1 0.17 at k=500) suggest substantial headroom, so the collection can support future comparisons as methods improve."],"supporting_citations":[{"why":"TREC Clinical Decision Support track, the main retrieval benchmark whose single-shot ICU narratives and lack of confirmed diagnoses DC3 is positioned against.","marker":"[19]"},{"why":"CLEF eHealth patient-centric retrieval task, which DC3 contrasts for lacking temporally ordered findings and confirmed target diagnoses.","marker":"[23]"},{"why":"MIMIC-III critical care database, cited to show existing open data is limited to intensive care indications and billing-derived ICD codes.","marker":"[6]"},{"why":"i2b2 obesity recognition task, an example of the narrow property-extraction focus of prior shared tasks.","marker":"[24]"},{"why":"i2b2 smoking-status task, another illustration that existing i2b2 corpora target a single patient property rather than broad diagnosis.","marker":"[25]"},{"why":"i2b2 heart-disease risk-factor task, limited to one disease class and therefore not a wide-coverage diagnostic benchmark.","marker":"[22]"},{"why":"Defines nDCG, the evaluation measure used for the patient-centric retrieval baselines.","marker":"[5]"},{"why":"National Academies report grounding the paper's motivating claim that diagnostic errors are frequent, harmful, and costly.","marker":"[13]"}],"fun_headline_variants":["31 tough diagnostic cases get dense PubMed relevance labels","DC3: 31 hard cases with full PubMed relevance judgments","Dense relevance labels for 31 diagnostic challenges across PubMed","Hard cases, dense labels: DC3 benchmark for diagnosis","31 case challenges with dense relevance over 27M PubMed abstracts"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole relevance labeling depends on the assumption that a paper whose title or abstract mentions one of the case's confirmed diagnoses is genuinely useful to the diagnosing physician, and that the proprietary medical named entity recognizer finds those mentions without meaningful error rates.","fun_headline_variants_meta":{"raw":{"variants":["31 tough diagnostic cases get dense PubMed relevance labels","DC3: 31 hard cases with full PubMed relevance judgments","Dense relevance labels for 31 diagnostic challenges across PubMed","Hard cases, dense labels: DC3 benchmark for diagnosis","31 case challenges with dense relevance over 27M PubMed abstracts"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000186,"raw_usage":{"total_tokens":1330,"prompt_tokens":958,"completion_tokens":372,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":574,"completion_tokens_details":{"reasoning_tokens":291}},"tokens_in":574,"tokens_out":372,"duration_ms":3901,"temperature":1.0,"reasoning_tokens":291,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T11:34:46.409107+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of documents the NER labeled non-relevant for a given DC3 case and have clinicians judge them for diagnostic usefulness; if a substantial fraction actually describe the confirmed diagnosis using synonyms or indirect terminology, or if a manual audit shows the NER misses many true mentions, the dense relevance judgments are not a reliable ground truth for benchmarking.","supporting_citations":[{"cited_title":"Overview of the trec 2014 clinical decision support track","cited_arxiv_id":null,"evidence_quote":"TREC Clinical Decision Support track, the main retrieval benchmark whose single-shot ICU narratives and lack of confirmed diagnoses DC3 is positioned against."},{"cited_title":"Overview of the share/clef ehealth evaluation lab 2013","cited_arxiv_id":null,"evidence_quote":"CLEF eHealth patient-centric retrieval task, which DC3 contrasts for lacking temporally ordered findings and confirmed target diagnoses."},{"cited_title":"Mimic-iii, a freely accessible critical care database","cited_arxiv_id":null,"evidence_quote":"MIMIC-III critical care database, cited to show existing open data is limited to intensive care indications and billing-derived ICD codes."},{"cited_title":"Recognizing obesity and comorbidities in sparse data","cited_arxiv_id":null,"evidence_quote":"i2b2 obesity recognition task, an example of the narrow property-extraction focus of prior shared tasks."},{"cited_title":"Identifying patient smoking status from medical discharge records","cited_arxiv_id":null,"evidence_quote":"i2b2 smoking-status task, another illustration that existing i2b2 corpora target a single patient property rather than broad diagnosis."},{"cited_title":"Identifying risk factors for heart disease over time: Overview of 2014 i2b2/uthealth shared task track 2","cited_arxiv_id":null,"evidence_quote":"i2b2 heart-disease risk-factor task, limited to one disease class and therefore not a wide-coverage diagnostic benchmark."},{"cited_title":"Cumulated gain-based evaluation of ir techniques","cited_arxiv_id":null,"evidence_quote":"Defines nDCG, the evaluation measure used for the patient-centric retrieval baselines."},{"cited_title":"National Academies Press, 2016","cited_arxiv_id":null,"evidence_quote":"National Academies report grounding the paper's motivating claim that diagnostic errors are frequent, harmful, and costly."}],"review_version":1}