{"id":"2f07e43e-c8ea-439c-bca1-07361345f492","arxiv_id":"2507.01049","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A DPR-based retrieval model trained on echocardiography reports outperforms BM25 and off-the-shelf embeddings on common cohort queries, but underperforms on rare-condition (out-of-distribution) searches.","lead":"This paper applies a neural search technique called Dense Passage Retrieval to find groups of patients from echocardiography reports in a hospital database. The authors show their custom-trained model beats standard search baselines on most of their tests, though it fails on rare conditions.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Evaluation reuses training queries and templated numeric queries, so the reported superiority may reflect query memorization rather than generalization to unseen clinical questions.","rationale":"The reader's weakest_assumption correctly identifies query memorization and templated numerical queries as the core threat to the central claim. I agree that this is load-bearing: the strongest evidence for the claim comes from held-out and numerical sets whose construction allows high scores through training-query reuse and template overlap. However, I would add that the label construction itself (exact statement occurrence) is a separate but related validity risk, since it defines relevance in a way that may not match clinical semantics and may disadvantage models that generalize beyond surface forms. The reader's proposed condition (release code/annotations, add statistical robustness, strengthen held-out queries) is appropriate. I do not see an internal inconsistency that would warrant rejection; the paper is transparent about its OOD failure and the reported numbers are plausible given the experimental design. Thus, the verdict remains CONDITIONAL, and my stress-test does not move the reader's judgment—hence UNCHANGED. The concrete test would settle whether the memorization concern actually degrades performance on genuinely unseen queries, which is the decisive check for the central claim.","tokens_in":11505,"tokens_out":3048,"duration_ms":36093,"concrete_test":"Build a truly held-out query set: take reports not used in training, extract their LV findings statements, and create paraphrases by clinicians (or by an LLM with templates disjoint from Table 2) that were never seen during training. Evaluate v0, v1, v2, BM25, ClinicalBERT, and Jina on this set with P@10/R-Precision, and report separately for the four subcategories excluded from training. If the custom model's advantage shrinks below the reported 0.82-0.99 range, the memorization concern lands. Also compute 95% bootstrap confidence intervals over the 100 numerical queries to check whether the v1/v2 numerical superiority over BM25 is statistically reliable.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that a custom DPR model outperforms baselines for echocardiography cohort retrieval rests on evaluation sets whose queries overlap with training in a way that may inflate scores. In Section 3.3.1, the held-out set uses 'the same set of queries used in the training set' with only new passages; the model has therefore seen the exact query strings during training, paired with different passages. High P@10 on such held-out passages can be achieved by memorizing query-to-condition mappings rather than by learning to match novel query phrasings. The four subcategories excluded from training are not reported separately, so we cannot tell whether the model generalizes to genuinely unseen query types. In Section 3.3.3, numerical evaluation queries are generated from templated expressions (Table 2) with random values, closely resembling the templates used to synthesize training triplets for v1/v2; the near-perfect scores are thus measured on the same distribution the model was trained on, not on free-text numerical questions. Additionally, the relevance labels are constructed by exact statement occurrence (the inverted index in Section 3.2), not by clinician judgment, so a passage that is clinically relevant but phrased differently is counted as negative. This can penalize models that generalize semantically and may advantage models that memorize surface forms. These concerns do not disprove the claim but directly weaken the external validity of the headline numbers.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper applies Dense Passage Retrieval (DPR) to patient cohort retrieval in echocardiography. The authors transform the Left Ventricle section of MIMIC-III echocardiography reports into query statements and use the Summary sections as passages, constructing an inverted index for relevance labels. They evaluate three custom-trained DPR variants (v0, v1, v2), BM25, ClinicalBERT, and Jina embeddings on four tasks: held-out passages, paraphrased queries, numerical LVEF retrieval, and out-of-distribution (OOD) conditions. They report that the custom models outperform baselines on the first two tasks, achieve near-perfect scores on the numerical task for v1/v2, and all fail on OOD, while BM25 performs best on OOD. They also propose a modification to Multiple Negatives Ranking loss that masks in-batch positives that are relevant to the anchor query.","tokens_in":11709,"tokens_out":4370,"duration_ms":49820,"significance":"If the reported results are valid, the paper makes a useful practical contribution: a systematic pipeline for converting an unstructured clinical EHR corpus into a query-passage retrieval dataset, a domain-specific dense retriever that outperforms off-the-shelf alternatives on several retrieval tasks, and a loss modification that addresses a subtle problem in cohort retrieval with multiple relevant passages. The paper is honest in reporting the OOD failure and in stating the limitations of the dataset. The weaknesses are concentrated in the evaluation design: the held-out set reuses training queries, the numerical evaluation queries come from the same template distribution used in training, and there are no statistical significance or uncertainty estimates. These issues weaken the external validity of the headline claim of 'superior performance' but do not invalidate the relative ordering of the models on the specific test splits.","major_comments":[{"comment":"The held-out set is described as containing 'the same set of queries used in the training set' with only new passages. This design tests generalization to new passages for query types already seen in training, but not generalization to novel query phrasings. Consequently, the high P@10 values in Table 3 (0.85 for v0) could largely reflect the model memorizing query-to-condition mappings rather than matching new query formulations. The four subcategories excluded from training are the only genuinely unseen query types, yet their results are not reported separately. Please report performance on those four subcategories separately, or construct a held-out set with queries not seen during training, to support the claim of superior generalization to clinical queries.","section":"§3.3.1"},{"comment":"The numerical evaluation queries are automatically generated from templated expressions shown in Table 2 (e.g., 'LVEF {sign} {mid_value}%') with random values, and the v1 training set is augmented with triplets created from the same temporary LVEF database and similar templates. Although the paper states that the evaluation queries were excluded from training, the distribution of the evaluation queries is essentially identical to the training distribution. The near-perfect P@10 scores of 0.98 and 0.99 for v1 and v2 are therefore expected for a model trained on that exact distribution. To support the claim of superior numerical retrieval, evaluate on free-text numerical queries written by clinicians or on templates that were not used in training; otherwise the numerical results should be described as in-distribution performance, not generalization.","section":"§3.3.3 and §4"},{"comment":"Relevance labels are constructed by exact statement occurrence via the inverted index: a passage is relevant only if it contains the exact statement (or one of its manually grouped variations) assigned to the query. A passage that expresses the same clinical condition using different wording is counted as negative. This labeling scheme penalizes models that generalize semantically and may advantage models that rely on surface-form matching. As a result, the absolute P@10 and R-Precision values in Table 3 are not directly interpretable as clinical retrieval accuracy. The paper should either add a small set of manual/clinical relevance judgments to calibrate the automatic labels, or explicitly discuss this limitation and its potential effect on the reported scores.","section":"§3.2 and §3.4"},{"comment":"All reported metrics are single-point estimates with no error bars, bootstrap confidence intervals, or significance tests. Given the small evaluation sets (100 numerical queries, 60 paraphrased queries, and a held-out set whose size is not stated), differences such as v0 vs. v1 on the held-out set (P@10 0.85 vs. 0.81) may be within noise. The paper should report variance across multiple training runs or bootstrap confidence intervals, and ideally paired significance tests for the main comparisons, to support the claim that the custom DPR model is 'superior' to the baselines.","section":"Table 3"}],"minor_comments":[{"comment":"The paper says the model was 'trained from scratch without any prior pretrained weights' but also that it used 'msmarco-bert-base-dot-v5 as architecture.' Since 'msmarco-bert-base-dot-v5' refers to a pretrained checkpoint, clarify whether the model was initialized with random weights (i.e., only the architecture is taken from that model) or with the pretrained checkpoint; the current wording is contradictory.","section":"§4"},{"comment":"The OOD evaluation uses conditions that occur only in single-digit numbers, and the trained models achieve P@10 of 0.00–0.06. Please report the number of OOD queries and passages used in the evaluation, as results at this scale are highly variable and a single query can shift the metric substantially.","section":"§3.3.2"},{"comment":"The model names in Table 3 are concatenated with the first metric values (e.g., 'echo-retriever-model v00.85'); add spacing for readability.","section":"Table 3"},{"comment":"The Limitations section states that 'comparisons were restricted to embedding-based models,' but BM25 is a lexical, non-embedding baseline. Please revise the wording to acknowledge that the comparison includes a traditional IR baseline.","section":"Limitations"},{"comment":"The paraphrased query set contains 60 queries. Given the small size, report the number of queries per subcategory and indicate whether the paraphrases were manually validated for medical correctness and preservation of intent.","section":"§3.3.4"}],"recommendation":"major_revision","confidential_remarks":"The central methodological idea—domain-specific DPR with a modified loss—is plausible, but the evaluation design currently overstates the generalization claim. The paper would be substantially strengthened by reporting the excluded-subcategory results, constructing a truly novel numerical query set, and adding uncertainty estimates. In addition, the 'first work to apply DPR for patient cohort retrieval in the echocardiography domain' claim should be checked against the broader clinical trial retrieval literature (e.g., DeepEnroll and related dense retrieval methods), which is not discussed in the related work."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nThis is a real applied contribution with a genuine caveat: the headline numbers are likely inflated by evaluation design that overlaps training. Worth a referee's time, but needs revision before I'd trust the claimed superiority.\n\nWhat's actually new: first DPR system for echocardiography cohort retrieval, built on EchoNotes/MIMIC-III, with a thoughtful pipeline for turning 130k LV statements into queries/passages. The loss modification—masking in-batch positives so they don't become false negatives in MNR—is sensible and addresses a real issue with multi-positive queries. The evaluation includes OOD and paraphrased tasks, and the authors honestly report that their models fail on OOD (v2 gets P@10 0.00). That transparency counts.\n\nThe soft spots are the ones in the stress-test, and they're real. The held-out set uses the same query strings as training, only with new passages; the model has memorized query-to-condition mappings, so P@10 of 0.85 on held-out is not the same as generalization to new query phrasings. The four excluded subcategories aren't broken out, so we can't see whether the model handles genuinely unseen query types at all. The numerical task is worse: training and evaluation queries come from the same templates (Table 2), so the 0.98–0.99 P@10 on numerical tasks mostly shows the model learned the template distribution, not that it can answer free-text numeric questions. The relevance labels are built by exact statement occurrence via the inverted index, which penalizes semantically equivalent phrasing—this can advantage models that memorize surface forms. No error bars or significance tests anywhere. The authors' Limitations section is candid about scope but doesn't mention the query-overlap issue.\n\nNone of this disproves the practical value of the pipeline or the relative ordering on these splits. A custom-trained DPR model beating BM25 on paraphrased queries (0.68 vs 0.26) is a meaningful result even if the gap on held-out is partly memorization. But the external validity of the reported superiority is substantially weaker than the abstract suggests.\n\nWho's it for: applied clinical IR researchers building cohort retrieval systems, who want a benchmark and a pipeline to adapt. Not a methods paper—the DPR architecture and loss are standard with a small twist. I'd send it to peer review, but flag major revision: release the annotations or at least query templates, add a truly held-out query set with clinician labels, report the excluded subcategories separately, and give error bars.\n\nI wouldn't cite it in my own work in the next year, but I'd bring it to reading group if someone in our group works on medical retrieval.\n\nHope this helps.","headline":"A useful applied DPR study for echo cohort retrieval, but evaluation overlaps training queries and the numbers overstate generalization.","tokens_in":12249,"tokens_out":2449,"would_cite":false,"duration_ms":25134,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A custom-trained dense passage retriever, built from echocardiography reports, outperforms BM25 and off-the-shelf clinical embeddings on patient cohort retrieval tasks.","keywords":["dense passage retrieval","patient cohort retrieval","echocardiography","electronic health records","multiple negatives ranking loss","BM25","LVEF","clinical information retrieval"],"falsifier":"Construct a fresh set of LVEF queries written by clinicians in natural language, with no overlap with the paper's templates, and compare the v1 and v2 models against BM25 on the same passage corpus. If P@10 drops to BM25's level, the reported numerical superiority is an artifact of template memorization. The same test can be run on paraphrased queries by generating them with a different LLM and prompt than the one used to create the training paraphrases.","tokens_in":11253,"feed_emoji":"🩺","tokens_out":5768,"duration_ms":51547,"temperature":0.7,"pith_summary":"The paper argues that patient cohort retrieval from echocardiography reports is better solved by a dense passage retriever trained from scratch on the domain's own query-passage pairs than by traditional lexical retrieval or off-the-shelf clinical embedding models. It builds a dataset from a 43,472-report intensive-care echocardiography collection by turning Left Ventricle findings into query statements and report summaries into passages, then evaluates on held-out, paraphrased, out-of-distribution, and numerical LVEF queries. On the non-OOD tasks the custom model reaches P@10 between 0.82 and 0.99, clearly above BM25 and ClinicalBERT. If these results hold, they establish a template for adapting DPR to other medical note domains with dense, condition-rich text.","feed_headline":"Custom retriever beats BM25 and ClinicalBERT on echo cohorts","feed_subtitle":"Trained on echocardiography reports, dense passage retrieval scores P@10 of 0.82–0.99 across three clinical query tasks","key_machinery":"The central object is a modified Multiple Negatives Ranking Loss applied to a shared bi-encoder DPR model. Standard MNR treats every other pair's positive passage as an in-batch negative, which is wrong when many queries share relevant passages; the paper sets the cosine similarity of non-anchor positives to minus infinity so they contribute no misleading gradient. Training triplets draw the positive and a hard negative from different subcategories of the same clinical condition, forcing the model to learn fine-grained distinctions, and numerical LVEF training is supported by a temporary database of extracted values, signs, and ranges.","core_discovery":"On its own terms, the paper claims that a custom-trained DPR embedding model is superior to traditional and off-the-shelf state-of-the-art methods for patient cohort retrieval in echocardiography. The authors convert unstructured echo reports into a query-passage corpus, then train a shared bi-encoder embedding model from scratch so that queries and report summaries are close in embedding space when the summary mentions the queried condition. On evaluation sets built to mimic clinical use, the custom model beats both BM25 and off-the-shelf clinical and general embedding models on held-out reports, paraphrased queries, and LVEF numerical queries: the numerical variant reaches P@10 of 0.99, the held-out variant reaches 0.85, and paraphrased-query performance reaches 0.68. The paper also reports that out-of-distribution rare conditions remain unsolved, with all dense variants scoring near zero while BM25 reaches 0.48.","pith_inferences":["The numerical evaluation uses queries generated from the same templates seen in training, so the 0.98-0.99 P@10 likely overstates performance on free-text numerical questions; varying the wording or adding distractors would test this.","The held-out set reuses training query statements with new passages, so part of the gain may come from memorized query-to-condition mappings rather than passage understanding; the paraphrased set is the cleaner evidence of generalization.","The OOD collapse suggests a hybrid system that combines lexical retrieval for rare terms with dense retrieval for common paraphrases may be the practical path, though the paper does not propose it.","If the approach is extended beyond the Left Ventricle to other anatomies, the condition hierarchy and hard-negative sampling would need to handle overlapping labels, possibly requiring a different loss."],"forward_implications":["Domain-specific DPR can replace term matching and general clinical embeddings for cohort retrieval when the same clinical condition is expressed in many surface forms.","The modified Multiple Negatives Ranking Loss, which masks other pairs' positives, makes DPR training viable when one query maps to many relevant passages.","Numerical cohort queries such as LVEF thresholds can be handled by combining heuristic value extraction with templated training queries, reaching near-perfect P@10 on the paper's evaluation.","The same dataset-construction and evaluation recipe transfers to other medical note domains that have structured findings plus free-text summaries.","Out-of-distribution rare conditions remain an open problem: lexical BM25 stays the best method there, and dense variants score at most 0.06 P@10."],"supporting_citations":[{"why":"Supplies the 43,472-report echocardiography dataset and its structured labels that seed the query-passage construction.","marker":"(Kwak et al., 2024)"},{"why":"Introduces the DPR dual-encoder formulation that the paper adapts to cohort retrieval.","marker":"(Karpukhin et al., 2020)"},{"why":"Defines Multiple Negatives Ranking Loss, which the paper modifies by masking in-batch positives.","marker":"(Henderson et al., 2017)"},{"why":"Provides the BERT architecture and masked-language-model pre-training used for the v2 variant.","marker":"(Devlin et al., 2019)"},{"why":"Is the probabilistic retrieval foundation behind the BM25 baseline that the central comparison must beat.","marker":"(Crestani et al., 1998)"},{"why":"Supplies ClinicalBERT, the evaluated off-the-shelf clinical embedding baseline.","marker":"(Huang et al., 2020)"},{"why":"Supplies Jina embeddings v2, the general-purpose long-context embedding baseline.","marker":"(Günther et al., 2024)"},{"why":"Motivates the quantity-aware LVEF parsing and numerical query generation.","marker":"(Almasian et al., 2024)"},{"why":"Provides the local LLM used to paraphrase training queries and cluster condition statements.","marker":"(Dubey et al., 2024)"}],"fun_headline_variants":["Custom DPR beats BM25 and ClinicalBERT on echo cohort retrieval","Echo cohort retrieval: custom DPR outperforms BM25 and ClinicalBERT","DPR for echo cohorts: custom model tops BM25 and ClinicalBERT","New DPR embedding wins on echo cohort retrieval, but rare cases fail"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the evaluation sets measure true clinical retrieval behavior rather than the model recognizing query templates it already saw in training.","fun_headline_variants_meta":{"raw":{"variants":["Custom DPR beats BM25 and ClinicalBERT on echo cohort retrieval","Echo cohort retrieval: custom DPR outperforms BM25 and ClinicalBERT","DPR for echo cohorts: custom model tops BM25 and ClinicalBERT","New DPR embedding wins on echo cohort retrieval, but rare cases fail"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000443,"raw_usage":{"total_tokens":2217,"prompt_tokens":890,"completion_tokens":1327,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":506,"completion_tokens_details":{"reasoning_tokens":1248}},"tokens_in":506,"tokens_out":1327,"duration_ms":10474,"temperature":1.0,"reasoning_tokens":1248,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T22:20:26.541725+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct a fresh set of LVEF queries written by clinicians in natural language, with no overlap with the paper's templates, and compare the v1 and v2 models against BM25 on the same passage corpus. If P@10 drops to BM25's level, the reported numerical superiority is an artifact of template memorization. The same test can be run on paraphrased queries by generating them with a different LLM and prompt than the one used to create the training paraphrases.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the 43,472-report echocardiography dataset and its structured labels that seed the query-passage construction."},{"cited_title":"is this document relevant?…probably","cited_arxiv_id":null,"evidence_quote":"Is the probabilistic retrieval foundation behind the BM25 baseline that the central comparison must beat."}],"review_version":1}