{"id":"0a80cf42-8a9d-481a-a721-c4c8bfbd8e4d","arxiv_id":"2505.24090","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"SearchAI maps natural language clinical queries to families of ICD-10 codes with a hierarchical similarity search, but its reported accuracy rests on a self-referential benchmark.","lead":"This paper presents SearchAI, a system that turns natural language clinical queries into sets of ICD-10 codes by traversing the coding hierarchy with token and similarity matching. It reports accuracy above 98% on two datasets, but the evaluation derives test queries from the very code descriptions the system searches, so the headline accuracy is not an independent measure.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 98% accuracy claim is unsupported: test queries are generated from the very ICD-10 descriptions that form the retrieval index, so the evaluation measures near-duplicate retrieval rather than realistic clinical search.","rationale":"The reader's weakest assumption correctly identifies the core problem: the test queries in Section 5.2 are generated from the ICD-10 code descriptions, so the query text and the target descriptions share vocabulary and structure. The strongest claim in Section 6, that SearchAI is more than 98% accurate and outperforms manual search, depends entirely on this evaluation. Because the default hierarchical predictor in Section 4.1 uses exact token matching, generating queries directly from descriptions makes the task closer to duplicate detection than to open-ended semantic search. The robustness experiment in Section 5.3 does not fix this; it paraphrases the descriptions using ChatGPT but still derives each query from the target text, so the lexical overlap remains high and no realistic clinical vocabulary is introduced. Additionally, the paper never reports a manual search baseline, so the 'outperforms the manual search approach' component of the claim has no experimental support at all. The production dataset is inaccessible and no code is released, which prevents independent reproduction. These are load-bearing flaws: if the evaluation were redone with realistic, independently labeled queries, the reported accuracy could drop substantially. My concern is not that the algorithmic ideas are necessarily wrong, but that the evidence as presented does not support the central performance claim. Therefore the reader's REJECT verdict is appropriate, and my review does not change it.","tokens_in":8749,"tokens_out":2632,"duration_ms":32933,"concrete_test":"Construct a held-out evaluation set of 1,000 real-world clinical search queries (from physician notes, EHR search logs, or queries written by clinicians who have not seen ICD-10 descriptions), have two certified coders independently assign all relevant ICD-10 codes for each query, and measure SearchAI's precision, recall, and F1 against the union of their labels. Also run a simple BM25 or TF-IDF baseline on the same queries. If SearchAI's F1 is not above 98% while the baseline is substantially lower, the current metric is inflated; if both are near 98%, the evaluation is not discriminating.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In Section 5.2, accuracy is computed by converting each ICD-10 code description into a query and checking whether SearchAI retrieves that code. Since the retrieval index contains those same descriptions, and the default matcher in Section 4.1 uses exact token overlap, a large portion of the reported accuracy can be achieved by trivial lexical matching. The robustness test in Section 5.3 paraphrases the same descriptions with ChatGPT, so it also starts from the target text and does not introduce the vocabulary shift, abbreviations, misspellings, or multi-condition phrasing of real clinician queries. There is no held-out set of realistic queries with independent gold labels, no precision/recall breakdown, no inter-rater agreement, and no comparison against manual search, even though Section 6 claims SearchAI outperforms manual search. Consequently the central 'more than 98% accurate' claim is not established by the reported experiments.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents SearchAI, a system for searching clinical data by converting natural language queries into families of ICD-10 codes. The main algorithmic contribution is a hierarchical predictor that traverses the ICD-10 code hierarchy, with variants that flatten the hierarchy and combine hierarchical with flat random search. Evaluations on a Medicare FFS dataset and a production dataset report high accuracy (above 98% for level-1 codes), robustness to paraphrased queries, low latency, and linear scalability. The paper claims that SearchAI outperforms manual search approaches in accuracy, robustness, performance, and scalability.","tokens_in":9065,"tokens_out":4454,"duration_ms":45677,"significance":"If the reported accuracy were obtained on realistic queries, a natural-language clinical search tool with roughly 110 ms per-query latency would be practically useful and could reduce manual coding effort. The paper also usefully distinguishes the one-to-many clinical search problem from the one-to-one code assignment problem. However, the evidence presented does not establish the central claim: the accuracy and robustness evaluations are self-referential, no comparison to manual search or to existing coding tools is reported, and no held-out set of real clinician queries is used. The system's pseudocode is described, but no code, data, or reproducible artifacts are provided. The significance of the contribution therefore remains unproven in the current manuscript.","major_comments":[{"comment":"The robustness experiment suffers from the same circularity. ChatGPT is prompted to paraphrase each level-1 code description while avoiding words from the original description, so the generated variations still start from the target text and share its semantic and lexical structure. This does not model the vocabulary shift, abbreviations, misspellings, or multi-condition phrasing typical of real clinician queries. The robustness claim would require testing on naturally occurring query variations, for example from search logs or elicited from domain experts, rather than only on paraphrases of the target descriptions.","section":"Section 5.3"},{"comment":"The conclusion states that SearchAI 'outperforms the manual search approach in terms of accuracy, robustness, performance, and scalability,' but no manual search baseline is reported anywhere in Section 5. The experiments compare only the authors' own algorithmic variants (default, flattened, hybrid). Without a manual-coding or existing-tool comparison, this claim is unsupported. At minimum, the authors should define a manual search protocol, measure its accuracy and latency on the same query set, and report inter-rater agreement among human coders.","section":"Section 6"},{"comment":"The evaluation omits several technical details needed to interpret the headline accuracy. The similarity threshold in the default/hybrid predictors and the weights for vague terms are free parameters, yet the paper does not state how they were set (e.g., by tuning on a validation set) or report sensitivity analysis. Also, the reported 98%+ figures appear to be for level-1 codes only (Figures 3a/3b), while Section 6 claims 'more than 98% accurate' without qualification. Please report accuracy per hierarchy level, provide overall accuracy over all returned codes, and include confidence intervals or significance tests for the differences between variants.","section":"Sections 4.1 and 5.2"}],"minor_comments":[{"comment":"Typographical errors in code ranges: '100-199' should be 'I00-I99' and '000-09A' should be 'O00-O9A'.","section":"Table 1"},{"comment":"Step 4 of the pseudocode refers to a 'defined threshold' for approximate matching, but the threshold is never defined; please specify how it is chosen.","section":"Section 4.1"},{"comment":"The latency metric is ambiguous: it is unclear whether Figures 5a/5b report average per-query latency or total time for all level-1 queries, and the axes are not labeled.","section":"Section 5.4"},{"comment":"The text says 'Figures 6a show' (grammar), and the FFS sample sizes (200, 400, 840) do not match the stated fractions 25%, 50%, and 100%; please clarify the dataset construction.","section":"Section 5.5"},{"comment":"The paper emphasizes 'generative AI,' but the hierarchical predictor described in Section 4 uses token-overlap matching and Levenshtein distance, with no generative component; this mismatch between the framing and the actual method should be addressed.","section":"Abstract and Section 4"},{"comment":"Reference [10] is cited for statistics about manual coding costs and ICD-9/ICD-10 comparisons, but the cited URL is a paper abstract page; please verify that the source supports these claims and provide a proper citation.","section":"References"}],"recommendation":"reject","confidential_remarks":"The manuscript reads as an industry technical report rather than a rigorous scientific study. The central accuracy claim is not supported due to the circular evaluation design, and the claimed comparison with manual search is entirely absent. Substantial new experiments with realistic queries and external baselines would be needed to make the paper acceptable for a database or health-informatics venue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper makes one point I think is right and useful: clinical data search is not the same as code assignment. It is one-to-many, and the ICD-10 hierarchy matters. The authors build a hierarchy-aware retrieval pipeline, with token promotion, flattening, and a hybrid random search. That is a reasonable engineering effort, and their internal comparisons show the variants beat the default predictor on their own metrics. The scalability and latency numbers are also sensible.\n\nThe problem is the accuracy evaluation. Section 5.2 generates queries by converting each ICD-10 code description into a question, and the expected answer is the same description stored in the index. That measures near-duplicate retrieval, not search. The robustness test paraphrases the same descriptions with ChatGPT, so it still starts from the target text. No realistic held-out queries, no manual search comparison (despite the conclusion claiming to outperform manual search), no confidence intervals, no precision/recall breakdown. The 98% figure is therefore not credible as stated.\n\nThere are also smaller issues: the 'generative AI' label is used loosely; the actual matching is mostly token overlap and Levenshtein distance. The production dataset is inaccessible, and no code is released, which makes reproducibility impossible.\n\nI don't think this is a wasted submission. The problem framing and the hierarchical traversal ideas could be useful to others working on ICD search. But the evidence does not support the claims. If this comes to peer review, I'd want to see a real evaluation with clinician-generated queries, baselines, and error analysis. As it stands, it's a systems paper with a promising approach and a flawed demonstration.\n\nRecommendation: it deserves a serious referee, because the problem is real and the approach is engaged. But I'd expect heavy revision before it is publishable.","headline":"A useful framing of one-to-many ICD search, but the 98% accuracy claim is unsupported by a self-referential evaluation.","tokens_in":9427,"tokens_out":2500,"would_cite":false,"duration_ms":29227,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SearchAI claims that natural-language clinical search can be made reliable by treating it as a one-to-many hierarchical lookup, mapping each query to a whole family of ICD-10 codes rather than a single code.","keywords":["clinical data search","ICD-10 codes","generative AI","hierarchical traversal","natural language query","one-to-many retrieval","healthcare informatics","Boolean predicate decomposition"],"falsifier":"Run SearchAI against a held-out set of queries written from scratch by practicing clinicians (not derived from code descriptions), score the returned code families against a gold standard assigned by certified medical coders, and check whether accuracy remains above 98%; if realistic-query accuracy drops materially, the central claim is refuted.","tokens_in":8563,"feed_emoji":"🩺","tokens_out":6602,"duration_ms":65877,"temperature":0.7,"pith_summary":"SearchAI is a proposed generative-AI system for searching clinical data, and this paper's central claim is that clinical data search is a one-to-many problem: a single natural-language query should return a family of ICD-10 codes, not one code. The authors build a pipeline that splits a query into Boolean filter predicates, translates each predicate into code comparisons, and runs a hierarchical predictor over the ICD-10 tree. Their main experimental result is that a hybrid traversal—top-down hierarchical search combined with a flat random search to pick a starting point—reaches roughly 98% accuracy on level-1 codes in both a public Medicare dataset and a production dataset, with about 110 ms worst-case per-query latency. The motivation is practical: manual coding takes about half an hour per hospital admission and coding mistakes cost an estimated $25 billion a year in the US, so reliable natural-language lookup would reduce administrative burden and support faster diagnosis.","feed_headline":"SearchAI maps plain-English queries to ICD codes at 98% accuracy","feed_subtitle":"One-to-many code lookups replace manual chart review, with per-query latency near 110 ms.","key_machinery":"The central object is the hierarchical predictor, a traversal algorithm over the ICD-10 code tree that is trained to respect parent-child relationships and return all applicable descendant codes for a query. The decisive variant is the hybrid predictor: it first attempts a top-down hierarchical search, then runs a flat random search across all levels; if the random search finds a better-scoring starting point, it returns that node and all its descendants. The other variants are the default token-matching predictor and the hierarchy-flattening refinement, which rewrites unreachable child nodes by adjusting their depth so every node becomes reachable. The predictor is coupled with a query-decomposition stage that turns natural language into Boolean filter predicates (AND, OR, NOT) and with database-specific compilation for execution.","core_discovery":"The paper argues that the right formulation of clinical search is not code assignment (one condition to one code) but code search (one query to a family of codes), and that this family must respect parent-child relationships in the coding hierarchy. On that basis, SearchAI decomposes natural-language questions into Boolean predicates and maps each predicate to comparison operators over the database. The load-bearing experimental result is the comparison of traversal strategies: the default hierarchical predictor scores 67.35% on the FFS dataset and 60% on the production dataset at level-1, hierarchy flattening improves this to approximately 99% and 98.3% respectively, and the hybrid variant reaches 98.63% and 98.6%. When queries are rephrased semantically, the hybrid-plus-flattened variant retains 79.86% accuracy on FFS and 88.23% on production. The paper concludes that SearchAI is more than 98% accurate and outperforms manual search in accuracy, robustness, performance, and scalability.","pith_inferences":["The robustness test rephrases descriptions with ChatGPT while banning original words, which is a strong proxy for paraphrase but not for free-form clinical language; a natural next experiment is to evaluate on real EHR search logs, where accuracy is likely to be lower than the reported 98%.","Because flattening is fitted to a specific database instance, the paper's design suggests a per-instance tuning cost; one could test a learned policy that decides when flattening helps, using multiple hospital datasets.","The same hierarchical predictor could be applied to LOINC and CPT code trees without re-architecting, since the algorithm only assumes a parent-child code hierarchy; an extension would compare its accuracy on lab-test and procedure codes."],"forward_implications":["A physician can type 'show sepsis patients' and receive the full set of relevant ICD-10 codes rather than a single code, making cohort discovery faster and less error-prone.","The roughly 30-percentage-point accuracy gain over the default traversal shows that encoding the hierarchy explicitly, rather than flattening it globally, is what enables reliable one-to-many lookup.","At about 110 ms per query in the worst case, the approach is interactive enough for point-of-care use.","Keeping all hierarchy paths reachable, as the flattening refinement does, prevents long-tail codes from being silently dropped during lookup."],"supporting_citations":[{"why":"Supplies the ICD-10 code ontology that SearchAI traverses and evaluates.","marker":"[5]"},{"why":"Documents the existing automated coding accuracy (76%) that motivates moving from assignment to search.","marker":"[8]"},{"why":"Documents a higher-accuracy automated coding recommendation (99.4%) that sets expectations for AI coding assistance.","marker":"[9]"},{"why":"Quantifies the manual coding burden (about 30 minutes per admission and $25 billion annual error cost) that SearchAI aims to reduce.","marker":"[10]"}],"fun_headline_variants":["SearchAI maps plain-English queries to ICD code families at 98% accuracy","One query, many codes: generative AI search for clinical data at 98%+","Hierarchical generative AI search for clinical data hits 98% accuracy","Clinical search AI outperforms manual and default lookup with 98% accuracy","SearchAI: generative AI for clinical data search yields 98%+ accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation builds test queries by converting each ICD-10 code description into a question, so the queries share vocabulary and sentence structure with the descriptions in the search index; real clinician queries are unlikely to align as tightly, and the accuracy claim depends on that alignment.","fun_headline_variants_meta":{"raw":{"variants":["SearchAI maps plain-English queries to ICD code families at 98% accuracy","One query, many codes: generative AI search for clinical data at 98%+","Hierarchical generative AI search for clinical data hits 98% accuracy","Clinical search AI outperforms manual and default lookup with 98% accuracy","SearchAI: generative AI for clinical data search yields 98%+ accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00065,"raw_usage":{"total_tokens":3022,"prompt_tokens":1028,"completion_tokens":1994,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":644,"completion_tokens_details":{"reasoning_tokens":1904}},"tokens_in":644,"tokens_out":1994,"duration_ms":16841,"temperature":1.0,"reasoning_tokens":1904,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:34:42.735385+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run SearchAI against a held-out set of queries written from scratch by practicing clinicians (not derived from code descriptions), score the returned code families against a gold standard assigned by certified medical coders, and check whether accuracy remains above 98%; if realistic-query accuracy drops materially, the central claim is refuted.","supporting_citations":[{"cited_title":"ICD Code Lists","cited_arxiv_id":null,"evidence_quote":"Supplies the ICD-10 code ontology that SearchAI traverses and evaluates."},{"cited_title":"Comparing Spark NLP for Healthcare and ChatGPT in Extracting ICD10-CM Codes from Clinical Notes","cited_arxiv_id":null,"evidence_quote":"Documents the existing automated coding accuracy (76%) that motivates moving from assignment to search."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents a higher-accuracy automated coding recommendation (99.4%) that sets expectations for AI coding assistance."},{"cited_title":"Few-shot ICD coding with knowledge transfer and evidence representation","cited_arxiv_id":null,"evidence_quote":"Quantifies the manual coding burden (about 30 minutes per admission and $25 billion annual error cost) that SearchAI aims to reduce."}],"review_version":1}