{"id":"6dced15e-021d-48d5-992e-2a6a6eabd936","arxiv_id":"2502.06252","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"CliniQ is a public EHR retrieval benchmark with 77,206 LLM-annotated relevance judgments, showing that BM25 is a strong baseline and that semantic matches drive dense-retriever gains.","lead":"Researchers built CliniQ, a public benchmark for searching electronic health records, with 1,246 queries and 77,206 relevance judgments drawn from 1,000 MIMIC-III discharge summaries. It lets teams compare retrieval methods and measure how well they handle synonyms, abbreviations, subtypes, and implied medical concepts.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Multi-Patient gold standard is incomplete by construction: chunks from patients without the queried code are scored as irrelevant, biasing all Multi-Patient metrics and model rankings.","rationale":"The central claim of the paper is that CliniQ provides the first publicly available, large-scale EHR retrieval benchmark supporting reliable evaluation in both Single-Patient and Multi-Patient settings. The Single-Patient component is well supported: for each patient, all queries in that patient's coding set are annotated across all chunks of that patient's note, and human evaluation on a sampled subset shows high agreement with medical experts. However, the Multi-Patient component is the load-bearing weak spot because its relevance judgments are incomplete by construction. Only queries assigned to a patient's structured codes are annotated for that patient's chunks; all other query-chunk pairs are implicitly treated as irrelevant. Given that MIMIC coding is known to be incomplete and does not capture every clinical mention, the Multi-Patient gold standard systematically misses relevant chunks. This is not merely a theoretical concern: it directly affects the headline metrics reported in Table 3 and the conclusions about which retrieval methods work best in the Multi-Patient scenario. Dense retrievers, which can match semantic paraphrases, are more likely to retrieve relevant chunks from patients without the corresponding billing code, and those retrievals are counted as errors. This can distort the reported ranking of BM25 against dense models, and the paper's own observation that 'general domain dense retrievers surprisingly outperform those designed for the medical domain' may partly reflect this annotation bias. The paper acknowledges the false-negative issue but asserts that established benchmarks also have false negatives; this comparison is imperfect because established test collections typically use pooling over many retrieval systems to build a more complete judgment set, whereas CliniQ's Multi-Patient judgments rely on a single structured-code source. The proposed test—adding judgments for a sample of unannotated chunks and recomputing metrics—would quantify the severity and determine whether the benchmark's Multi-Patient results are trustworthy. Given that the Single-Patient setting and the semantic-match analysis are still valuable, the appropriate disposition is to keep the reader's CONDITIONAL verdict, with the condition being that the Multi-Patient incompleteness be measured and either corrected or clearly caveated in the released benchmark.","tokens_in":19912,"tokens_out":4956,"duration_ms":42889,"concrete_test":"Take the 100 most frequent disease and procedure queries from CliniQ. For each query, randomly sample 200 chunks from patients whose structured codes do NOT include that query, and have GPT-4o (with the same prompt as in Figure 3) judge relevance for these unannotated pairs; for a smaller subset of, say, 10 queries, also obtain judgments from two medical experts. Compute the false-negative rate: the fraction of these currently unannotated chunks that are actually relevant. Then re-compute Multi-Patient MRR, NDCG@10, and recall@100 for BM25, NV-Embed-v2, and RRF after adding the newly judged relevant chunks to the gold standard and re-scoring. If the model rankings or the reported recall advantage of BM25 shifts by more than 5 relative points, the Multi-Patient benchmark results as presented are not reliable without a more complete annotation procedure.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.3 states: 'we only annotate queries assigned to the patient in the coding systems Q_i, rather than all queries Q, for each note due to cost constraints.' Consequently, in the Multi-Patient setting, for a query q, chunks from any patient whose structured codes do not include q are never judged, and all such chunks are counted as irrelevant. MIMIC ICD and prescription codes are billing-derived and incomplete; the paper itself notes this in Limitations: 'the incompleteness of ICD labels introduces unavoidable false negatives.' The positive set for each query is therefore restricted to chunks from coded patients, while relevant mentions in non-coded patients—e.g., mentions in past medical history or conditions documented but not billed—are systematically excluded. This makes the reported MRR, NDCG@10, and recall@100 in Table 3 scores against a partial gold standard rather than a complete one. The bias is not neutral across methods: dense retrievers that surface semantically related chunks from non-coded patients are penalized as false positives, while BM25, which tends to retrieve exact-string chunks from coded patients, is comparatively advantaged. The paper's defense that established benchmarks also contain false negatives is not fully persuasive because TREC-style collections use pooling over many systems, whereas here the negative labels are generated from a single incomplete billing-code source, so the false-negative rate is likely higher and unquantified. The headline Multi-Patient results and the conclusions drawn from them (e.g., dense vs. sparse ranking, drug retrieval performance) are therefore not dependable until the severity of this incompleteness is measured.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces CliniQ, a publicly released EHR retrieval benchmark built from 1,000 MIMIC-III discharge summaries, chunked into 16,550 passages. Queries are derived from ICD-9 disease codes, ICD-9 procedure codes, and prescription labels, yielding 1,246 unique queries. Relevance is annotated at chunk level using exact string matching plus GPT-4o judgments, with match types classified as string, synonym, abbreviation, hyponym, or implication; the full dataset contains 77,206 positive relevance judgments. The benchmark supports two settings: Single-Patient Retrieval and Multi-Patient Retrieval. The authors evaluate BM25, UMLS-based query expansion, several open-source dense retrievers, a proprietary embedding model, and reciprocal rank fusion, and report that BM25 is a strong baseline, general-domain dense retrievers outperform biomedical-domain ones, implication matches are the hardest, and fusion improves results.","tokens_in":20167,"tokens_out":13333,"duration_ms":123843,"significance":"If the annotation quality and completeness claims hold, CliniQ is a valuable community resource: it is, to my knowledge, the first large-scale public EHR retrieval benchmark with fine-grained semantic match categories, and it includes a human evaluation on 5,221 judgments showing very high agreement with GPT-4o. The release of code and patient identifiers for corpus reconstruction is a practical and appropriate way to handle MIMIC data-use restrictions. The benchmark also provides a broad, reproducible comparison of sparse and dense retrievers, which is useful for the community. However, the Multi-Patient relevance judgments are incomplete by construction, and this directly affects the headline Multi-Patient rankings and the claim that BM25 is a strong baseline in that setting. The Single-Patient portion and the semantic match analysis are less affected, but the Multi-Patient results need substantially more support before the benchmark can be relied upon as a fair evaluation of the two settings.","major_comments":[{"comment":"The Multi-Patient gold standard is incomplete in a way that is not neutral across methods. Section 3.3 states that semantic-match judgments are produced only for queries assigned to a patient in the coding systems Q_i, rather than for all queries Q, and Section 4.1 defines Multi-Patient Retrieval as retrieving for every query over all 16,550 chunks. Therefore, for a query q, chunks from patients whose structured codes do not include q are never judged for semantic relevance and are silently treated as irrelevant in Table 3. MIMIC ICD and prescription codes are billing-derived and incomplete, as the paper itself notes in the Limitations, so a documented condition in a non-coded patient's note can easily be missed. This creates an unquantified false-negative rate that is likely higher for dense retrievers, which surface semantically related text, than for BM25, which retrieves lexically matching chunks from coded patients. Because the Multi-Patient MRR, NDCG@10, and recall@100 in Table 3 are computed against this partial gold standard, the central conclusion that BM25 is a strong Multi-Patient baseline relative to dense retrievers is not established. The defense in the Limitations that all retrieval benchmarks contain false negatives is not fully persuasive: TREC-style collections use pooling over many systems, whereas here negative labels are generated from a single incomplete billing-code source without pooling. Please either (a) construct the Multi-Patient judgments by pooling candidate chunks from multiple retrieval systems and annotating them, (b) re-annotate a random sample of non-assigned patient-query pairs to estimate the false-negative bias and show that model rankings are robust to it, or (c) explicitly restrict the Multi-Patient claims to the annotated patient-query pairs and avoid presenting the current scores as complete relevance-based metrics.","section":"Section 3.3, Section 4.1, Table 3"},{"comment":"Table 1 reports 77.3 relevant judgments per patient in Single-Patient Retrieval, which projects to about 77.3k over 1,000 notes, while the Multi-Patient total is 77,206. The Single-Patient match-type subcounts (29.2, 24.8, 3.0, 4.3, 15.9) are exactly the Multi-Patient totals (29,149, 24,798, 3,039, 4,288, 15,932) divided by 1,000. This is surprising because Section 4.1 says the Single-Patient query set is filtered to queries with at least one positive chunk in that note, whereas Section 3.3's global string matching should add extra string-match positives in the Multi-Patient setting beyond the within-patient positives. The two totals therefore cannot coincide in the way the table suggests. Please clarify how the Single-Patient column was computed, or correct the table; as written, it conflates the two settings and undermines confidence in the reported dataset statistics.","section":"Section 5.1, Table 1"}],"minor_comments":[{"comment":"There is a typo: 'Among dense retrivers with a parameter size less than 7B' should read 'Among dense retrievers with fewer than 7B parameters'.","section":"Section 5.2"},{"comment":"The heading 'Query type assessment' refers to 'Simple-Patient Retrieval' in the body text; this should be 'Single-Patient Retrieval'.","section":"Section 5.4"},{"comment":"The phrase 'Cohen's Kappa efficient of 1' should be 'Cohen's kappa coefficient of 1'.","section":"Section 2.1.3"},{"comment":"The caption reports averages of MRR, NDCG, and MAP for Single-Patient Retrieval and averages of MRR, NDCG@10, and recall@100 for Multi-Patient Retrieval; since these metrics have different scales, the figure should state whether the components were normalized before averaging.","section":"Figure 1"}],"recommendation":"major_revision","confidential_remarks":"The main risk is the Multi-Patient label completeness issue. If the authors can add a robustness check, such as re-annotating a sample of the currently unjudged patient-query pairs or using a pooled annotation approach, the contribution would be substantially stronger. The Single-Patient benchmark and the semantic match analysis are promising and likely valuable to the community even if the Multi-Patient comparisons require revision. The self-citations in the related work are background references and do not appear to create circularity. The paper's format appears to be a conference template without an actual conference target; this should be cleaned up before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing you should know first: this is the first publicly released, large-scale EHR retrieval benchmark on MIMIC-III. 1,246 queries, 77,206 relevance judgments, and a five-way semantic match taxonomy (string, synonym, abbreviation, hyponym, implication) that actually lets you see what dense retrievers buy you. That alone makes it worth reading.\n\nThe construction is sensible. Queries come from structured ICD disease/procedure codes and prescription labels, which is reproducible; the corpus is chunked discharge summaries. The human evaluation on 5,221 annotations (141 chunks, 10 notes) shows GPT-4o agreeing with ground truth at 0.985–0.995 on relevance and match type, comparable to two senior M.D. annotators. That supports the annotation quality claim for the sampled part. The experiments are comprehensive: BM25 with and without UMLS expansion, eight dense retrievers, OpenAI embedding, and RRF. The observation that general-domain dense retrievers beat biomedical ones, and that BM25 is stubbornly competitive, is a useful data point for the community.\n\nNow the soft spots. The Multi-Patient gold standard is incomplete by construction. Section 3.3 says they only annotate each chunk against that patient’s own coded queries, not all queries. So for a query like 'diabetes mellitus', chunks from patients whose structured codes don't include diabetes are never judged and silently counted as irrelevant. Since MIMIC ICD/prescription codes are billing-derived and incomplete—the paper's own limitation says so—this creates systematic false negatives. The stress-test is right that this isn't neutral: dense retrievers that surface semantically related chunks from non-coded patients get penalized, while BM25, which tends to pull exact-string chunks from coded patients, is advantaged. The paper's defense that all benchmarks have false negatives doesn't fully hold, because TREC uses pooling across many systems; here the negative labels come from a single incomplete source. So the headline Multi-Patient rankings in Table 3 should be read as provisional, not as ground truth. The authors partly acknowledge this in the Limitations, but they don't quantify the bias.\n\nA smaller concern: 77k labels rest on a 1% human sample. The agreement is high, but per-note variance isn't reported, and the sample is only 10 notes. Negation handling is also absent, which can matter for clinical queries. Minor relative to the main issue.\n\nThe Single-Patient setting, where every query–chunk pair is annotated, is the solid core. That part deserves serious attention. The Multi-Patient half needs a quantified false-negative analysis, or a pooled annotation subset, before its results can be trusted.\n\nWho's this for? Anyone building or evaluating clinical IR systems, especially dense retrieval in medical domains. It deserves a serious referee—the resource is valuable, the flaws are documented and addressable, and the Single-Patient analysis is a real contribution. I'd take it to reading group and would cite it.","headline":"CliniQ is a genuinely useful new EHR retrieval resource, but the Multi-Patient half runs on an incomplete gold standard the paper itself concedes; the Single-Patient half is the solid part.","tokens_in":20740,"tokens_out":2664,"would_cite":true,"duration_ms":22123,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper introduces CliniQ, a public benchmark for electronic health record retrieval built on MIMIC-III with 1,246 queries, 77,206 labeled relevance judgments, and a five-way match-type taxonomy that exposes the semantic gap.","keywords":["Electronic Health Record Retrieval","Test Collection","Semantic Gap","Dense Retrieval","BM25","MIMIC-III","Relevance Judgment","LLM Annotation"],"falsifier":"Re-annotate a random sample of patient-note chunks against the full global query set of 1,246 queries instead of only the queries assigned to those patients. If a non-trivial fraction of previously unlabeled pairs turns out to be relevant, the multi-patient recall figures are inflated and the completeness assumption fails. A second check would be to collect a fresh human-annotated holdout beyond the 141 chunks in the paper and test whether GPT-4o's match-type labels maintain the reported agreement.","tokens_in":19737,"feed_emoji":"🏥","tokens_out":7121,"duration_ms":55673,"temperature":0.7,"pith_summary":"CliniQ is a public benchmark for retrieving information from electronic health records, built from 1,000 de-identified discharge summaries in MIMIC-III and 1,246 queries derived from ICD disease codes, ICD procedure codes, and prescription labels. It provides 77,206 chunk-level relevance judgments, each labeled as a string match or one of four semantic match types — synonym, abbreviation, hyponym, or implication — making it an order of magnitude larger than previous EHR retrieval test collections. The paper argues that this benchmark enables standardized, fair evaluation of both single-patient retrieval and multi-patient retrieval, and makes the semantic gap between queries and clinical text measurable for the first time. A sympathetic reader would care because prior benchmarks were proprietary, small, or never released, leaving EHR retrieval systems without a common yardstick.","feed_headline":"EHR retrieval gets its first large public benchmark","feed_subtitle":"CliniQ releases 77,206 labeled judgments and a five-way match taxonomy to test clinical search systems.","key_machinery":"The load-bearing object is the CliniQ test collection and its annotation protocol. Discharge summaries are split into 100-word overlapping chunks; queries are obtained by aggregating ICD-9 disease codes (mapped to three-digit ancestors), ICD-9 procedure codes, and cleaned prescription names; exact string search finds literal matches; and GPT-4o with a chain-of-thought prompt classifies every remaining query–chunk pair as irrelevant or as one of four semantic match types. The two retrieval settings (single-patient and multi-patient) define the evaluation regimes, and the match-type taxonomy turns the semantic gap into five concrete categories. A human-evaluated subset of 141 chunks with 5,221 annotations shows GPT-4o in near-perfect agreement with two senior medical annotators, which is the evidentiary basis for trusting the automated labels.","core_discovery":"The central claim is that a large-scale public EHR retrieval benchmark can be built by combining a public corpus with LLM-based annotation, and that such a benchmark changes what can be seen about retrieval systems. CliniQ is that benchmark: 1,000 MIMIC-III discharge summaries chunked into 16,550 segments, queried with 1,246 coded entities, and labeled with 77,206 relevance judgments plus a five-way match-type taxonomy. Using it, the paper shows that BM25 is a strong baseline, that dense retrievers' advantage over BM25 is almost entirely due to semantic matches, that implication matches are the hardest category for every tested method, and that general-domain dense retrievers outperform those trained for the biomedical domain. The paper also validates the annotation pipeline against human experts, reporting near-perfect agreement on both relevance and match type.","pith_inferences":["The per-match-type breakdown could serve as a diagnostic suite for future systems: a model that improves implication matching without sacrificing string matching would be identifiable from CliniQ's labeled subsets, a finer signal than a single pooled score.","The annotation recipe — exact matching plus LLM refinement over a public coded corpus — generalizes to other note types and coding systems, although CliniQ itself only covers discharge summaries.","If adopted, CliniQ's match-type labels could be used directly as training signal, e.g., as targets for contrastive learning that rewards semantic equivalence rather than only surface similarity, which the paper does not explore.","The benchmark's reliance on MIMIC-III access means its long-term availability depends on that dataset remaining accessible; porting the pipeline to MIMIC-IV would test the collection method's reproducibility."],"forward_implications":["BM25 with UMLS query expansion should remain the reference baseline in future EHR retrieval studies, since it stays competitive with dense retrievers, especially in multi-patient settings.","Dense retrieval research for EHRs should focus on semantic match types, because that is where dense models earn their advantage; implication match is the hardest category for every method tested.","General-domain dense retrievers outperforming biomedical ones indicates that EHR text is out of distribution for current biomedical embeddings, motivating EHR-specific training data.","RRF-style fusion of sparse and dense retrieval gives large gains on both shallow and deep metrics, making hybrid retrieval a practical direction for clinical deployment.","Drug queries, which are short and mostly literal, favor lexical matching; dense retrievers need specific work to preserve fine-grained string information."],"supporting_citations":[{"why":"Supplies the MIMIC-III discharge summaries and structured ICD/prescription codes that CliniQ's corpus and query set are built from.","marker":"[20]"},{"why":"The prior attempt to build an EHR retrieval benchmark on MIMIC-III that did not release annotations, defining the gap CliniQ fills.","marker":"[33]"},{"why":"The TREC Medical Records Track, the previously most-used EHR retrieval test collection whose small scale (85 queries) motivates CliniQ's larger query set.","marker":"[54]"},{"why":"A prior EHR retrieval test collection with human expert annotations (5,815 positive pairs) that CliniQ compares against in scale and method.","marker":"[55]"},{"why":"Chain-of-Thought prompting, the technique used to elicit GPT-4o's relevance judgments and match-type classifications.","marker":"[56]"},{"why":"Defines BM25, the sparse retriever that serves as the strong baseline against all dense retrievers.","marker":"[40]"},{"why":"MedCPT, a biomedical dense retriever used as a medical-domain baseline in the evaluation.","marker":"[16]"},{"why":"BMRetriever, the LLM-based biomedical retriever series included as medical-domain dense baseline.","marker":"[60]"}],"fun_headline_variants":["CliniQ: EHR retrieval benchmark with semantic match types","77K relevance judgments power new EHR retrieval test","BM25 matches dense retrievers on EHR benchmark","Semantic matches reveal EHR retrieval strengths","New public benchmark for EHR retrieval with five match types"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the relevance judgments are complete enough to serve as ground truth: for each patient's notes, only the queries originally coded for that patient were annotated, so any chunk that is actually relevant to a different query is silently treated as irrelevant in the multi-patient evaluation.","fun_headline_variants_meta":{"raw":{"variants":["CliniQ: EHR retrieval benchmark with semantic match types","77K relevance judgments power new EHR retrieval test","BM25 matches dense retrievers on EHR benchmark","Semantic matches reveal EHR retrieval strengths","New public benchmark for EHR retrieval with five match types"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000968,"raw_usage":{"total_tokens":4133,"prompt_tokens":977,"completion_tokens":3156,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":593,"completion_tokens_details":{"reasoning_tokens":3084}},"tokens_in":593,"tokens_out":3156,"duration_ms":21073,"temperature":1.0,"reasoning_tokens":3084,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T16:14:08.487963+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-annotate a random sample of patient-note chunks against the full global query set of 1,246 queries instead of only the queries assigned to those patients. If a non-trivial fraction of previously unlabeled pairs turns out to be relevant, the multi-patient recall figures are inflated and the completeness assumption fails. A second check would be to collect a fresh human-annotated holdout beyond the 141 chunks in the paper and test whether GPT-4o's match-type labels maintain the reported agreement.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the MIMIC-III discharge summaries and structured ICD/prescription codes that CliniQ's corpus and query set are built from."},{"cited_title":"Miller, Yanjun Gao, Matthew M","cited_arxiv_id":null,"evidence_quote":"The prior attempt to build an EHR retrieval benchmark on MIMIC-III that did not release annotations, defining the gap CliniQ fills."},{"cited_title":"Voorhees","cited_arxiv_id":null,"evidence_quote":"The TREC Medical Records Track, the previously most-used EHR retrieval test collection whose small scale (85 queries) motivates CliniQ's larger query set."},{"cited_title":"Hersh, Steven Bedrick, and Hongfang Liu","cited_arxiv_id":null,"evidence_quote":"A prior EHR retrieval test collection with human expert annotations (5,815 positive pairs) that CliniQ compares against in scale and method."},{"cited_title":"BMC Medical Informatics and Decision Making 21 (2021)","cited_arxiv_id":null,"evidence_quote":"Chain-of-Thought prompting, the technique used to elicit GPT-4o's relevance judgments and match-type classifications."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"MedCPT, a biomedical dense retriever used as a medical-domain baseline in the evaluation."}],"review_version":1}