{"id":"ce78f2cf-4de6-4877-a2dc-5d136a87ea0b","arxiv_id":"2501.02727","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"HiRMed, a hierarchical RAG-based medical test recommender, is claimed to reach 92.3% coverage and 88.7% accuracy, but the supporting dataset, code, and evaluation details are not provided.","lead":"This paper describes HiRMed, a system that combines a three-layer hierarchy, retrieval-augmented generation, and a fine-tuned language model to recommend medical tests from patient symptoms. The authors report higher coverage and accuracy and a lower miss rate than two simpler baselines, but provide no code, data, error bars, or independent clinical validation.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported superiority of HiRMed is not interpretable because §IV-C defines coverage and miss rate without fixing recommendation-list length or score threshold; the Table II gaps could reflect output volume rather than reasoning quality.","rationale":"The most load-bearing condition for the central claim is not whether the fine-tuning labels match the evaluation labels, but whether the evaluation can distinguish a better recommender from a more permissive one. §IV-C defines coverage and miss rate in ways that reward longer output lists, and §IV-D1 reports only aggregate point estimates with no list lengths, thresholds, confidence intervals, or test-set sizes. This is prior to the label-leakage issue: even with fully independent expert labels, the comparison is uninterpretable absent a fixed operating point. The reader's weakest assumption (training/eval label independence) is a real and unresolved concern, but I see the uncontrolled output policy as more basic and more directly checkable. A single computational re-analysis with matched list lengths and precision-recall curves would settle it. I therefore do not change the reader's REJECT verdict: the paper does not currently support its superiority claim. No machine-checked proofs, code, or data are provided, so independent verification is impossible from the manuscript.","tokens_in":7221,"tokens_out":3923,"duration_ms":41515,"concrete_test":"Require the authors to make available, for every test-set case, the ranked recommendation lists and the score cutoff/list-length used by HiRMed, Flat-RAG, and TVS, and to recompute Table II at matched list lengths (top-1, top-3, top-5, top-10) and as precision-recall curves. If HiRMed still dominates coverage, accuracy, and miss rate at fixed list lengths, the claim survives; if the advantage shrinks, reverses, or is explained by list length, the reported performance gap is an artifact.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Central claim: Section IV-D1, Table II. The metrics in §IV-C are not anchored to a decision threshold. Coverage Rate is 'proportion of relevant diagnostic tests included in recommendations' and Miss Rate is 'proportion of critical tests not recommended.' Both are monotonically improvable by adding more tests to the ranked list. Accuracy is the percentage of recommended tests deemed appropriate—a precision-like measure—so it penalizes over-recommendation, but no list length, score cutoff, or precision-recall trade-off is reported. The comparison with Flat-RAG and TVS therefore conflates recommendation quality with output policy: a method that emits 20 tests per case can beat a method that emits 5 on coverage and miss rate even if its ranking is worse. The same confound affects the ablation drops in Table IV, because removing memory or a layer may change how many items the system outputs. With no fixed operating point, the central claim that hierarchical RAG improves medical test recommendation is not established by the numbers.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes HiRMed, a three-layer hierarchical RAG-based medical test recommendation system that routes a patient query from a root department-level analysis, through department-specific reasoning, to item-level test ranking, using GPT-o1 for reasoning and a fine-tuned LLaMA3.2-3B weighted model. The manuscript reports results on 125,000 outpatient visits, claiming that HiRMed achieves a coverage rate of 92.3%, accuracy of 88.7%, miss rate of 2.1%, and clinical relevance score of 4.3/5, outperforming Flat-RAG and traditional vector similarity. An ablation study in Table IV attributes these gains to the memory module, the department layer, and the dual knowledge base. The core claim is that hierarchical RAG with memory augmentation and a fine-tuned ranking model improves medical test recommendation.","tokens_in":7365,"tokens_out":3638,"duration_ms":38382,"significance":"If the reported results were interpretable, the paper would offer a plausible architecture for combining hierarchical reasoning with retrieval-augmented generation in medical decision support, and the ablation study would provide useful evidence about which components matter. The problem is relevant and the design is clearly described. However, the manuscript provides no code or data, no statistical characterization, and the evaluation methodology has two load-bearing weaknesses: the metrics are not anchored to a decision threshold, and the independence of training and evaluation labels is not established. These issues prevent the paper from supporting its central claim in its current form.","major_comments":[{"comment":"Coverage Rate and Miss Rate are defined without fixing the recommendation-list length or a score threshold. Both metrics are monotonically improvable by adding more tests to the ranked list, while Accuracy is a precision-like measure that penalizes over-recommendation. With no operating point, precision-recall trade-off, or list-length control, the differences in Table II may reflect output volume rather than ranking or reasoning quality. The same confound applies to the ablation drops in Table IV, since removing the memory module or department layer could change how many items the system emits. The paper needs to report fixed operating points, precision-recall curves, or list-length-matched comparisons to make the comparisons meaningful.","section":"§IV-C, Table II"},{"comment":"The fine-tuned LLaMA3.2-3B weight model is trained on physician-annotated relevance scores (§III-B3), while evaluation compares recommendations against expert-validated test orders and a 12-clinician panel (§IV-C and §IV-D3). The manuscript does not establish that the training annotations and the evaluation labels are independent. If the same guidelines, standards, or clinicians produced both, then the headline numbers partly measure how well the model reproduces its training signal, not its clinical utility. The paper must report the provenance of the 500 reviewed cases, how they were sampled, who the 12 clinicians were, whether they were blinded to the system's output and to the original physician orders, and whether they were involved in creating the training annotations.","section":"§III-B3, §IV-C, §IV-D3"},{"comment":"The experimental results are reported as single point estimates with no confidence intervals, error bars, significance tests, or any description of the train/test split. The claim in §IV-D1 that HiRMed 'significantly outperforms' Flat-RAG and TVS is therefore unsupported. With 500 reviewed cases, a coverage rate of 92.3% has a 95% confidence interval of roughly ±2.3 percentage points, so some of the reported differences could overlap with those of the baselines. The paper should provide uncertainty quantification and, ideally, inter-rater reliability for the clinician panel.","section":"§IV-A, §IV-D1"}],"minor_comments":[{"comment":"Table IV uses both negative and positive numbers under 'Performance Drop (%)' but the text in §IV-D4 refers to 'a 15.3% increase in miss rate' and '11.6% decrease in coverage rate.' It would be clearer to state explicitly whether these are percentage-point changes or relative changes.","section":"§IV-D4, Table IV"},{"comment":"The model name 'GPT-O1' should be written consistently; the correct product name is 'OpenAI o1' or 'GPT-01' depending on the intended notation.","section":"Throughout"},{"comment":"The manuscript says the LLaMA model was trained on historical outpatient data but never describes the training/validation split for the fine-tuning data, the number of epochs, or the evaluation set construction. A short experimental setup subsection should clarify this.","section":"§IV-A"},{"comment":"The text references Figure 1a and 1b, but the figures are not embedded in the submitted text. The authors should ensure that the architecture diagram and workflow figure are included in any final submission.","section":"Figure 1"}],"recommendation":"reject","confidential_remarks":"The reader's report and the stress-test note both identify the same load-bearing weaknesses: the evaluation metrics are not threshold-anchored, and the independence of training and evaluation labels is unverified. I agree with those assessments. Even if the architecture has merit, the current manuscript's experimental section cannot support the headline claims without a substantial reworking of the evaluation protocol, and the paper would likely need new experiments rather than just clarifications. The reported numerical results should not be cited as evidence for the method's effectiveness in their present form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a reasonable system paper with a clear architecture and a worthwhile problem, but the headline empirical claims are not verifiable as reported. The central argument—that hierarchical RAG with memory beats flat retrieval for medical test recommendation—is plausibly true but not established by the numbers here.\n\nWhat is new: the specific combination of a three-layer RAG hierarchy, a memory mechanism, and a fine-tuned LLaMA reranker for test recommendation is a genuine synthesis, and the ablation study is a sensible way to probe the components. The paper also documents a 125k-visit outpatient dataset, even though it is not released. That is worth credit.\n\nThe soft spots, in order of severity. First, the stress-test concern is correct: coverage rate and miss rate are recall-like and monotonically improvable by emitting more tests, while accuracy is precision-like. With no list length, score threshold, or precision-recall trade-off reported, the Table II gaps could just mean HiRMed outputs a longer list than the baselines. The same confound affects the ablation drops in Table IV. Second, the circularity worry is real. The LLaMA model is fine-tuned on physician-annotated relevance scores (Section III-B3), and the evaluation uses expert-validated test orders plus a 12-clinician panel (Sections IV-C and IV-D3). The paper never shows that these labels come from independent sources, so the accuracy numbers may partly measure how well the model reproduces its training signal. Third, there are no error bars, no statistical tests, and no description of the train/test split. The baselines are also weak—only vector similarity and a single-layer RAG, with no standard recommender or clinical decision-support comparators. Finally, calling the system \"tree-based\" overstates what is a fixed three-layer pipeline; there is no dynamic branching or search.\n\nNone of these are fatal to the underlying idea. The architecture is coherent, the writing is clear, and the problem matters. But the paper is not ready for publication as a claims-driven empirical study. Who is this for? Researchers working on RAG pipelines for clinical decision support, and the evaluation-protocol lessons apply more broadly.\n\nIf I were an editor, I would send this to peer review rather than desk reject, because the system design is legitimate and the flaws are fixable in principle. The reviewers should demand data/code release, a fixed operating point or PR curves, an independent evaluation label source, and stronger baselines. If the authors cannot release data, the empirical claim stays unverifiable and the paper should be reframed as a systems description, not a superiority claim.","headline":"A plausible three-layer RAG pipeline for medical test recommendation, but the headline numbers are not interpretable because no decision threshold or list length is fixed and the evaluation may not be independent of the training labels.","tokens_in":7914,"tokens_out":1819,"would_cite":false,"duration_ms":19961,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"HiRMed claims that routing medical test recommendations through a three-layer tree of retrieval-augmented reasoning steps beats flat retrieval on coverage, accuracy, miss rate, and clinical relevance.","keywords":["Medical Test Recommendation","Retrieval-Augmented Generation","Tree-based Recommendation","Medical Reasoning","Large Language Models","Healthcare Decision Support","Hierarchical Reasoning","Memory-Augmented Reasoning"],"falsifier":"Audit the 500 reviewed cases behind the clinical relevance score for overlap with the physician-annotated training labels of the ranking model, and check whether any of the 12 clinicians contributed to those labels; if overlap exists, the reported 92.3%, 88.7%, 2.1%, and 4.3 results would be ambiguous. A cleaner test is to recompute all metrics on a held-out set scored by clinicians who had no part in creating training annotations.","tokens_in":6976,"feed_emoji":"🩺","tokens_out":11029,"duration_ms":99945,"temperature":0.7,"pith_summary":"HiRMed is a three-layer tree-structured recommendation system that decides which medical tests an outpatient should receive. At each node of the tree—root, department, and item—it does not just match patient symptoms to test descriptions by vector similarity; it retrieves relevant medical knowledge and performs a step of language-model reasoning to refine diagnostic hypotheses. The authors claim that this hierarchical structure, together with a memory of the reasoning history and a fine-tuned ranking model, lifts coverage to 92.3%, accuracy to 88.7%, cuts the miss rate to 2.1%, and earns a clinical relevance score of 4.3 out of 5 on a dataset of 125,000 outpatient visits. If true, the practical payoff is that a machine could help clinicians order the right tests and miss fewer critical ones, while leaving a traceable reasoning path behind each recommendation.","feed_headline":"Tree-structured medical AI misses only 2.1% of critical tests","feed_subtitle":"Stepwise reasoning with retrieved medical knowledge beats flat retrieval on coverage, accuracy, and clinical relevance.","key_machinery":"The carrying mechanism is a three-level tree in which every node performs its own retrieval-augmented generation: it pulls the most relevant passages from a vector-indexed knowledge base of roughly 50,000 clinical guidelines and protocols, lets a language model reason about symptom–disease relationships and diagnostic urgency, and lets a separately fine-tuned ranking model assign weights to candidate next steps. The root layer consults department-level knowledge to route the case, the department layer consults specialty-specific test knowledge to narrow hypotheses, and the item layer consolidates flagged symptoms in memory and issues the final weighted test list. The dual-layer knowledge base and the node-to-node memory are what turn the tree from a static decision structure into a mechanism that revises and refines recommendations as reasoning proceeds.","core_discovery":"On its own terms, the paper's central discovery is that replacing a one-step similarity match with a hierarchy of retrieval-augmented reasoning steps changes the quality of medical test recommendations. HiRMed routes a case from a root layer that identifies likely departments, through a department layer that narrows hypotheses with specialty knowledge, to an item layer that produces a weighted test list; each layer retrieves from one of two knowledge bases and reasons with a large language model, and a memory mechanism carries context forward across nodes. Against Flat-RAG and traditional vector similarity, the paper reports that HiRMed wins on all four metrics: coverage 92.3% versus 84.7% and 72.8%; accuracy 88.7% versus 82.4% and 71.5%; miss rate 2.1% versus 5.8% and 10.6%; clinical relevance score 4.3 out of 5 versus 3.7 and 3.2. Ablation results in the paper attribute part of the gain to memory, part to the department layer, and the largest share to the dual-layer knowledge base.","pith_inferences":["A testable extension the paper leaves implicit is to apply the same tree-RAG-with-memory pattern to treatment recommendation or differential diagnosis, since the hierarchy itself is not test-specific.","The reported results mix the tree architecture with a particular high-capacity language model and a fine-tuned ranker; swapping in a smaller or open-weight model would separate architectural gains from model-capacity effects.","A prospective deployment test would run HiRMed on new outpatient cases alongside the ordering physician and compare its ranked list with the physician's final orders, using an evaluation panel that never saw training labels."],"forward_implications":["A hierarchical RAG tree with memory is a workable alternative to flat retrieval for medical test ordering: the reported gap on miss rate alone (2.1% versus 5.8% for Flat-RAG) is the safety-relevant margin.","The ablation results imply that each architectural choice pays: removing memory costs 8.2 points of coverage, removing the department layer costs 11.6, and collapsing the dual knowledge base costs 13.7.","Because every recommendation passes through explicit reasoning steps, clinicians can inspect and contest the path by which a test was recommended.","Consistent department-level results (coverage above 90% in cardiology, endocrinology, and gastroenterology) suggest the method transfers across specialties with different diagnostic protocols."],"supporting_citations":[{"why":"Introduces retrieval-augmented generation, the core mechanism HiRMed places at each tree node.","marker":"[6]"},{"why":"Applies RAG to symptom-to-disease mapping without hierarchy, defining the gap HiRMed fills.","marker":"[7]"},{"why":"Supplies the dynamic hierarchical reasoning and knowledge-graph interaction that motivates HiRMed's tree-plus-memory design.","marker":"[11]"},{"why":"Represents the patient-similarity diagnostic prediction approach against which vector-similarity baselines are drawn.","marker":"[8]"},{"why":"Demonstrates the standard patient-similarity prediction paradigm that traditional vector-similarity baselines build on.","marker":"[9]"},{"why":"Systematic review of patient similarity methods that anchors the traditional retrieval baseline.","marker":"[10]"},{"why":"Shows the value of incorporating domain clinical knowledge into text models, supporting HiRMed's dual-layer knowledge base.","marker":"[14]"}],"fun_headline_variants":["Tree-based RAG cuts critical test misses to 2.1%","HiRMed: hierarchical RAG reasoning drops miss rate to 2.1%","Stepwise RAG reasoning cuts medical test misses to 2.1%","Hierarchical RAG beats flat retrieval on coverage and miss rate","Medical test AI with tree reasoning: 2.1% miss rate"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the physician annotations used to fine-tune the ranking model came from a source independent of the expert-validated test orders and the 12-clinician panel that produced the evaluation scores; otherwise the headline numbers partly measure how well the system reproduces its own training labels rather than its clinical value.","fun_headline_variants_meta":{"raw":{"variants":["Tree-based RAG cuts critical test misses to 2.1%","HiRMed: hierarchical RAG reasoning drops miss rate to 2.1%","Stepwise RAG reasoning cuts medical test misses to 2.1%","Hierarchical RAG beats flat retrieval on coverage and miss rate","Medical test AI with tree reasoning: 2.1% miss rate"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000407,"raw_usage":{"total_tokens":2109,"prompt_tokens":938,"completion_tokens":1171,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":554,"completion_tokens_details":{"reasoning_tokens":1083}},"tokens_in":554,"tokens_out":1171,"duration_ms":9454,"temperature":1.0,"reasoning_tokens":1083,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:06:09.921246+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Audit the 500 reviewed cases behind the clinical relevance score for overlap with the physician-annotated training labels of the ranking model, and check whether any of the 12 clinicians contributed to those labels; if overlap exists, the reported 92.3%, 88.7%, 2.1%, and 4.3 results would be ambiguous. A cleaner test is to recompute all metrics on a held-out set scored by clinicians who had no part in creating training annotations.","supporting_citations":[{"cited_title":"Retrieval- augmented generation for knowledge-intensive nlp tasks,","cited_arxiv_id":null,"evidence_quote":"Introduces retrieval-augmented generation, the core mechanism HiRMed places at each tree node."},{"cited_title":"Drlk: Dynamic hierarchical reasoning with language model and knowl- edge graph for question answering,","cited_arxiv_id":null,"evidence_quote":"Supplies the dynamic hierarchical reasoning and knowledge-graph interaction that motivates HiRMed's tree-plus-memory design."},{"cited_title":"Patient similarity in prediction models based on health data: A scoping review,","cited_arxiv_id":null,"evidence_quote":"Demonstrates the standard patient-similarity prediction paradigm that traditional vector-similarity baselines build on."},{"cited_title":"Incorporating domain knowledge into natural language inference on clinical texts,","cited_arxiv_id":null,"evidence_quote":"Shows the value of incorporating domain clinical knowledge into text models, supporting HiRMed's dual-layer knowledge base."}],"review_version":1}