{"id":"c67e67d7-8845-4c6b-b05c-9cb8c974989a","arxiv_id":"2412.15232","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"An unsupervised graph-based ranking method, partial matching, and ontological expansion improve biomedical document retrieval precision for concept-centric queries, but not for generic ones.","lead":"This paper extends a biomedical literature search system that represents each paper as a small graph, adding unsupervised ranking, partial matching, and ontology-based query expansion. On concept-centric TREC Precision Medicine queries, the new ranking improves precision over the unranked system and roughly matches BM25, but on generic COVID-19 queries it falls behind.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Ignoring unjudged TREC documents in Section 4.3 inflates precision/nDCG; the headline 'higher precision' claim needs re-evaluation with standard TREC scoring.","rationale":"The reader's weakest_assumption correctly identifies the unjudged-document handling as load-bearing. I considered two other concerns: the abstract attributes precision gains to ontological rewriting when Table 1 shows precision drops when ontology is added (e.g., PM2020 P@10 drops from 0.55 to 0.52; PM2017-19 nDCG@10 drops), and the evaluation is reported only on topics with translation score >0.9. Both are real, but they affect the framing and generalizability rather than the core measurements: the unjudged-document protocol affects the validity of every precision and nDCG number in Tables 1 and 2. If the evaluation is fixed and the numbers survive, the paper is a useful, unsupervised contribution with shared code; if not, the central claim collapses. Credit is due for releasing code and results, and the recall comparisons are more robust, but the precision claim is not. Therefore the verdict remains CONDITIONAL pending a re-run with standard TREC evaluation.","tokens_in":17349,"tokens_out":4131,"duration_ms":34924,"concrete_test":"Recompute all P@k and nDCG@k values in Tables 1 and 2 using the official TREC qrels, treating every retrieved unjudged document as non-relevant, and apply the same protocol to Native BM25. If the reported differences between Partial Match + GraphRank/+Ontology and Native BM25 shrink or reverse, the 'higher precision' claim is not supported; if the differences persist, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing concern is the evaluation protocol in Section 4.3: the authors 'decided to ignore unjudged documents' after observing that their graph-based matching retrieved many unjudged documents (e.g., 44 at rank 20 and 113 at rank 30 for PM2020). TREC pooled judgments do not mark unjudged documents as irrelevant; excluding them from the denominators of P@k and nDCG@k inflates those metrics for any system that retrieves many unjudged documents, which is exactly the graph-based system. Native BM25, by contrast, uses term-based retrieval closely aligned with the original pooling, so it is likely to have fewer unjudged documents to drop. With different fractions of unjudged documents removed per system, the reported precision and nDCG comparisons are not on equal footing. The recall@1000 numbers (e.g., 0.86 vs 0.79 on PM2020) are less affected, but the precision and nDCG gains that motivate the claim of 'higher precision due to... ontological rewriting' may be artifacts of this asymmetric treatment. Re-running with standard TREC evaluation (treating unjudged as non-relevant or using bpref) is required before the central claim can be accepted.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper extends an existing graph-based biomedical document retrieval system with GraphRank, an unsupervised method for ranking documents that match narrative query graphs, and adds query relaxation through partial matching and ontological rewriting. The system is evaluated on the TREC Precision Medicine 2017-2020 benchmarks and TREC-COVID 2020, with comparisons against BM25 reranking and native BM25 retrieval. The central claim is that the combination of graph-based ranking and query relaxation yields higher precision and recall for concept-centric biomedical queries, with code, results, and topic-wise evaluations made publicly available.","tokens_in":17602,"tokens_out":8023,"duration_ms":75911,"significance":"If the evaluation issues are resolved, this is a useful contribution to graph-based information retrieval. The work is grounded in a real, large-scale digital library system (37M documents), requires no training data, and is fully unsupervised. The authors share their code and detailed topic-wise results, which aids reproducibility. The paper also provides a fair treatment of limitations by showing degraded performance on TREC-COVID, where concepts are not well supported. However, the central quantitative claims currently rest on a non-standard and likely biased evaluation protocol, so the significance is conditional on a corrected evaluation.","major_comments":[{"comment":"The decision to ignore unjudged documents when computing precision and nDCG is not a standard TREC evaluation and is asymmetric across systems. As the authors note, their graph-based matching retrieves many unjudged documents (44 at rank 20 and 113 at rank 30 on PM2020), whereas native BM25, being term-based and aligned with the original pooling, is likely to retrieve far fewer. Since unjudged documents are not marked irrelevant, dropping them from the denominators inflates P@k and nDCG for the graph-based system relative to BM25. This directly affects the abstract's claim of 'higher precision.' The authors should re-run the evaluation treating unjudged documents as non-relevant (or using bpref) and report both versions; if the conclusions change, the claims must be revised.","section":"Section 4.3, Tables 1 and 2"},{"comment":"Results are reported only on the subset of topics that could be translated into graph queries with a translation score above 0.9 (e.g., 26/30 for PM2017, 45/50 for PM2018, 37/40 for PM2019, and about half of TREC-COVID). The abstract's unqualified statement that the extensions 'improve the system further so that users can retrieve results with higher precision and higher recall' overstates the evidence, since the evaluation excludes a substantial fraction of benchmark topics. The conclusion in Section 5 is appropriately conditional ('If queries were concept-centric and the system knew those concepts'), but the abstract should match that framing or the paper should report full-topic results with appropriate caveats.","section":"Section 4.2, Table 2, and the abstract"},{"comment":"The evaluation uses wildcard predicates between the searched concepts, because the TREC Precision Medicine topics do not provide relation information. This means the benchmark tests a relaxed version of the system in which any interaction between concepts counts, rather than the intended narrative query graph setting where users specify predicates such as 'treats' or 'inhibits.' The predicate-specific components of GraphRank, especially the predicate specificity in Eq. (3), are therefore not actually exercised in the evaluation. The paper should either evaluate on data with explicit predicates (or manually assigned plausible predicates) or discuss more thoroughly how the wildcard-predicate assumption affects the validity of the central claim for the real system.","section":"Section 4.1 (Query Translation for benchmarks)"}],"minor_comments":[{"comment":"The sentence preceding Eq. (7) says the relational similarity is computed 'by averaging all edge neighbors' tf-idf, coverage, and confidence scores,' but Eq. (7) is a sum with no normalization by the number of neighbor edges. Please align the text and the formula (or normalize the sum) so the method is reproducible as described.","section":"Section 3.2, Eq. (7)"},{"comment":"The TREC-COVID rows list 'Topics (25/50)', but the text in Section 4.3 states that '24 out of 50 topics had a translation score above 0.9 and contained at least two concepts.' Please correct this numerical inconsistency.","section":"Section 4.3, Table 2"},{"comment":"No significance tests are reported. Given that several metric differences in Tables 1 and 2 are small (e.g., nDCG@10 differences of 0.01–0.03), the authors should report per-topic significance (e.g., paired bootstrap or permutation test) or explicitly state that the observed differences are not tested for significance.","section":"Section 4.3 (generally)"},{"comment":"The heading 'PM2017-2029' contains a typo; it should be 'PM2017-2019'.","section":"Section 4.3 heading"}],"recommendation":"major_revision","confidential_remarks":"The main blocker is the unjudged-documents evaluation protocol; this is a methodological issue, not an issue of intent, and the authors are transparent about their choice. The paper would be acceptable after a corrected evaluation and a more carefully qualified abstract. The paper is more of a systems/application contribution than a theoretical IR advance, but it is within the scope of the journal and the technical-report format is appropriate for reproducibility-focused work."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a competent, honest extension of the authors' own narrative-query-graph retrieval system. The new pieces—GraphRank (unsupervised, graph-only ranking), partial-match relaxation, and ontology-based upward expansion—are genuinely useful for the digital-library setting they target, and they evaluate on five TREC collections with shared code and detailed topic-wise plots. That is more than many system papers do.\n\nThe main contribution is showing that a graph-only, completely unsupervised ranking can be competitive with BM25 reranking on the same graph-retrieved set, and that partial matching plus ontology expansion roughly doubles recall@1000 on PM2020 (0.31 to 0.86) versus the old exact-match system. That is a real result for a deployed 37M-document service.\n\nThe soft spots are mostly about the evaluation. Section 4.3 says they 'decided to ignore unjudged documents' when computing precision and nDCG. TREC pooling doesn't mark unjudged as non-relevant, and the graph matcher retrieves far more unjudged documents than BM25 does. Dropping them asymmetrically inflates the graph system's precision and nDCG relative to BM25. The recall numbers are less sensitive, but the headline 'higher precision' cannot be assessed from the current tables. They should rerun with bpref or treating unjudged as irrelevant, and report on all topics, not only those with translation score >0.9. Also, the abstract's 'higher precision ... due to ontological rewriting' is not backed by the tables: ontology expansion consistently lowers P@10. The hand-set weights and predicate specificity are acknowledged and fine for a technical report, but without significance tests the PM2017-2019 comparisons to BM25 are suggestive, not conclusive.\n\nWho is this for? Someone building graph-based retrieval on entity-relation document representations, or a digital library team considering unsupervised ranking. It deserves a serious referee—the system is real, the methods are clearly specified, and the limitations are honestly discussed—but the precision claims need a corrected evaluation before they can be taken at face value. I'd send it to review, with the unjudged-document issue as the main request.","headline":"A solid engineering paper on ranking graph-based medical document retrieval, but its precision claims are undermined by the unjudged-document handling and the abstract overstates what the data show.","tokens_in":18088,"tokens_out":2977,"would_cite":false,"duration_ms":210476,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that graph-based document retrieval can beat keyword baselines in both precision and recall when ranking matched subgraphs and relaxing exact-match queries, all without supervision.","keywords":["graph-based ranking","narrative query graphs","biomedical document retrieval","query relaxation","ontological rewriting","unsupervised ranking","TREC Precision Medicine","document graphs"],"falsifier":"Re-run the PM2020 evaluation while treating every unjudged retrieved document as non-relevant (or judging a random sample of them); if Partial Match + Ontology + GraphRank no longer beats native BM25 in P@10 and nDCG, the paper's precision and recall advantage is an artifact of the disregarded unjudged documents.","tokens_in":17137,"feed_emoji":"🧬","tokens_out":7796,"duration_ms":70036,"temperature":0.7,"pith_summary":"The paper tries to establish that graph-based document retrieval need not end at exact-match results: once each document is represented as a graph of concept interactions, the system can score how well a matched subgraph reflects the query, and can relax the query in controlled ways to find more documents. It contributes GraphRank, an unsupervised ranking built from extraction confidence, tf-idf-style concept weights, coverage, and neighborhood similarity; Partial Matches; and ontological rewriting that generalizes concepts upward. The central empirical result is that on the TREC Precision Medicine 2020 benchmark, partial matching plus ontology expansion achieves Recall@1000 of 0.86 versus 0.79 for native BM25, while GraphRank raises precision at top ranks. A sympathetic reader would care because this offers a path to precise, explainable, and training-free retrieval for concept-centric scientific search.","feed_headline":"Graph ranking lifts recall past BM25 on biomedical search","feed_subtitle":"Partial matching plus ontology expansion reaches 0.86 recall on TREC-PM2020.","key_machinery":"The central object is the fragment, a subgraph of a document graph mapped isomorphically to a narrative query graph. GraphRank scores each fragment as the product of its translation fidelity and a weighted sum of four normalized similarities: minimum extraction confidence, minimum tf-idf of edges, minimum concept coverage, and relational similarity from neighboring edges; the document's score is its best fragment. Partial matches are ranked below all full matches, and ontological rewriting scores generalized concepts by inverse path length in the ontology, so broader concepts contribute with lower weight.","core_discovery":"The paper's claim is that exact-match graph retrieval can be turned into an effective ranked retrieval mode by scoring matched fragments rather than treating all matches as equally relevant. On the TREC-PM2020 benchmark, combining partial matching with ontological expansion reaches Recall@1000 of 0.86 against 0.79 for native BM25, and GraphRank improves P@10 over unranked full matches while staying unsupervised. The authors do not claim superiority over learned systems; they claim that their graph-based digital library system can now offer competitive ranking and higher recall on concept-centric queries, and that this is explainable because each score component corresponds to an inspectable property of the extracted document graph.","pith_inferences":["The fragment-scoring scheme, with its minimum-over-edges composition, could be ported to other structured document representations, for instance neural graph embeddings, as a query-conjunctiveness regularizer.","Replacing the Jaccard translation score with a semantic similarity measure would likely recover some of the recall gap observed on TREC-COVID, where the vocabulary is the bottleneck.","The inverse-path-length similarity used for ontological rewriting suggests a natural stopping rule for expansion: stop generalizing when the summed similarity budget falls below a threshold.","If the unjudged-document issue is corrected, the measured gains may shrink; a hybrid that switches to BM25 when translation confidence is low follows directly from the paper's suggested fallback mode."],"forward_implications":["The discovery system can now return ranked result lists without maintaining a separate text index, since all ranking signals come from the graph representation.","Partial matching increases recall beyond exact full matches, while full matches always outrank partial ones, preserving the precision of exactly matching documents.","Ontological expansion substantially improves recall on concept-centric queries, for example from 0.60 to 0.84 on PM2017, by retrieving documents that mention a more general concept.","GraphRank is unsupervised and uses equal weights by default, so it can be deployed in domains without training data and matches BM25 reranking performance on PM2020.","The approach is limited to concept-centric queries; generic information needs such as school closings during the pandemic are not translated well, and native BM25 outperforms it on TREC-COVID."],"supporting_citations":[{"why":"Defines the narrative query graph formalism and the discovery system that this paper extends with ranking and relaxation.","marker":"[11]"},{"why":"Supplies the statement extraction method that turns documents into the graph representation being scored.","marker":"[10]"},{"why":"Defines the TREC-PM2020 benchmark and its relevance judgments used for the main evaluation.","marker":"[26]"},{"why":"Provides the TREC-COVID benchmark that demonstrates the approach's limitations on generic queries.","marker":"[25]"},{"why":"Provides the TREC-PM2017 benchmark used in the evaluation.","marker":"[28]"},{"why":"Provides the TREC-PM2018 benchmark used in the evaluation.","marker":"[27]"},{"why":"Provides the TREC-PM2019 benchmark used in the evaluation.","marker":"[29]"},{"why":"Supplies the implementation used to compute the BM25 baselines.","marker":"[15]"},{"why":"Provides the concept annotations used in building the document graphs.","marker":"[33]"}],"fun_headline_variants":["Graph ranking outperforms BM25 on biomedical recall","Ontology rewriting boosts graph retrieval recall","Unsupervised ranking makes graph queries effective","Partial matching plus ontology beats BM25 recall","Biomedical graph search with ranked matches"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes that unjudged documents retrieved by the graph system can be dropped from precision and nDCG calculations, even though TREC pooling does not mark them irrelevant, which can inflate the reported gains over BM25.","fun_headline_variants_meta":{"raw":{"variants":["Graph ranking outperforms BM25 on biomedical recall","Ontology rewriting boosts graph retrieval recall","Unsupervised ranking makes graph queries effective","Partial matching plus ontology beats BM25 recall","Biomedical graph search with ranked matches"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000258,"raw_usage":{"total_tokens":1533,"prompt_tokens":846,"completion_tokens":687,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":462,"completion_tokens_details":{"reasoning_tokens":631}},"tokens_in":462,"tokens_out":687,"duration_ms":7647,"temperature":1.0,"reasoning_tokens":631,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T20:59:55.472231+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the PM2020 evaluation while treating every unjudged retrieved document as non-relevant (or judging a random sample of them); if Partial Match + Ontology + GraphRank no longer beats native BM25 in P@10 and nDCG, the paper's precision and recall advantage is an artifact of the disregarded unjudged documents.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the TREC-PM2020 benchmark and its relevance judgments used for the main evaluation."},{"cited_title":"Voorhees, William R","cited_arxiv_id":null,"evidence_quote":"Provides the TREC-PM2017 benchmark used in the evaluation."},{"cited_title":"Voorhees, William R","cited_arxiv_id":null,"evidence_quote":"Provides the TREC-PM2018 benchmark used in the evaluation."},{"cited_title":"Voorhees, William R","cited_arxiv_id":null,"evidence_quote":"Provides the TREC-PM2019 benchmark used in the evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the implementation used to compute the BM25 baselines."}],"review_version":1}