Pith. sign in

REVIEW 3 major objections 6 minor 11 references

Searching Clinical Data Using Generative AI

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read SearchAI claims that natural-language clinical search can be made reliable by treating it as a one-to-many hierarchical lookup, mapping each query to a whole family of ICD-10 codes rather than a single code.

desk verdict A useful framing of one-to-many ICD search, but the 98% accuracy claim is unsupported by a self-referential evaluation. read the letter →

arxiv 2505.24090 v1 pith:227WR3VO submitted 2025-05-30 cs.DB cs.AI

Karan Hanswadkar , Anika Kanchi , Shivani Tripathi , Shi Qiao , Rony Chatterjee , Alekh Jindal This is my paper · ORCID
classification cs.DBcs.AI
keywords clinicaldatasearchICD-10codesgenerativeAIhierarchicaltraversalnaturallanguagequeryone-to-manyretrievalhealthcareinformaticsBooleanpredicatedecomposition
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SearchAI is a proposed generative-AI system for searching clinical data, and this paper's central claim is that clinical data search is a one-to-many problem: a single natural-language query should return a family of ICD-10 codes, not one code. The authors build a pipeline that splits a query into Boolean filter predicates, translates each predicate into code comparisons, and runs a hierarchical predictor over the ICD-10 tree. Their main experimental result is that a hybrid traversal—top-down hierarchical search combined with a flat random search to pick a starting point—reaches roughly 98% accuracy on level-1 codes in both a public Medicare dataset and a production dataset, with about 110 ms worst-case per-query latency. The motivation is practical: manual coding takes about half an hour per hospital admission and coding mistakes cost an estimated $25 billion a year in the US, so reliable natural-language lookup would reduce administrative burden and support faster diagnosis.

What carries the argument

The central object is the hierarchical predictor, a traversal algorithm over the ICD-10 code tree that is trained to respect parent-child relationships and return all applicable descendant codes for a query. The decisive variant is the hybrid predictor: it first attempts a top-down hierarchical search, then runs a flat random search across all levels; if the random search finds a better-scoring starting point, it returns that node and all its descendants. The other variants are the default token-matching predictor and the hierarchy-flattening refinement, which rewrites unreachable child nodes by adjusting their depth so every node becomes reachable. The predictor is coupled with a query-decomposition stage that turns natural language into Boolean filter predicates (AND, OR, NOT) and with database-specific compilation for execution.

What would settle it

Run SearchAI against a held-out set of queries written from scratch by practicing clinicians (not derived from code descriptions), score the returned code families against a gold standard assigned by certified medical coders, and check whether accuracy remains above 98%; if realistic-query accuracy drops materially, the central claim is refuted.

Watch

Extended reading notes

Core claim

The paper argues that the right formulation of clinical search is not code assignment (one condition to one code) but code search (one query to a family of codes), and that this family must respect parent-child relationships in the coding hierarchy. On that basis, SearchAI decomposes natural-language questions into Boolean predicates and maps each predicate to comparison operators over the database. The load-bearing experimental result is the comparison of traversal strategies: the default hierarchical predictor scores 67.35% on the FFS dataset and 60% on the production dataset at level-1, hierarchy flattening improves this to approximately 99% and 98.3% respectively, and the hybrid variant reaches 98.63% and 98.6%. When queries are rephrased semantically, the hybrid-plus-flattened variant retains 79.86% accuracy on FFS and 88.23% on production. The paper concludes that SearchAI is more than 98% accurate and outperforms manual search in accuracy, robustness, performance, and scalability.

Load-bearing premise

The evaluation builds test queries by converting each ICD-10 code description into a question, so the queries share vocabulary and sentence structure with the descriptions in the search index; real clinician queries are unlikely to align as tightly, and the accuracy claim depends on that alignment.

Editorial extensions

If this is right

  • A physician can type 'show sepsis patients' and receive the full set of relevant ICD-10 codes rather than a single code, making cohort discovery faster and less error-prone.
  • The roughly 30-percentage-point accuracy gain over the default traversal shows that encoding the hierarchy explicitly, rather than flattening it globally, is what enables reliable one-to-many lookup.
  • At about 110 ms per query in the worst case, the approach is interactive enough for point-of-care use.
  • Keeping all hierarchy paths reachable, as the flattening refinement does, prevents long-tail codes from being silently dropped during lookup.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The robustness test rephrases descriptions with ChatGPT while banning original words, which is a strong proxy for paraphrase but not for free-form clinical language; a natural next experiment is to evaluate on real EHR search logs, where accuracy is likely to be lower than the reported 98%.
  • Because flattening is fitted to a specific database instance, the paper's design suggests a per-instance tuning cost; one could test a learned policy that decides when flattening helps, using multiple hospital datasets.
  • The same hierarchical predictor could be applied to LOINC and CPT code trees without re-architecting, since the algorithm only assumes a parent-child code hierarchy; an extension would compare its accuracy on lab-test and procedure codes.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper presents SearchAI, a system for searching clinical data by converting natural language queries into families of ICD-10 codes. The main algorithmic contribution is a hierarchical predictor that traverses the ICD-10 code hierarchy, with variants that flatten the hierarchy and combine hierarchical with flat random search. Evaluations on a Medicare FFS dataset and a production dataset report high accuracy (above 98% for level-1 codes), robustness to paraphrased queries, low latency, and linear scalability. The paper claims that SearchAI outperforms manual search approaches in accuracy, robustness, performance, and scalability.

Significance. If the reported accuracy were obtained on realistic queries, a natural-language clinical search tool with roughly 110 ms per-query latency would be practically useful and could reduce manual coding effort. The paper also usefully distinguishes the one-to-many clinical search problem from the one-to-one code assignment problem. However, the evidence presented does not establish the central claim: the accuracy and robustness evaluations are self-referential, no comparison to manual search or to existing coding tools is reported, and no held-out set of real clinician queries is used. The system's pseudocode is described, but no code, data, or reproducible artifacts are provided. The significance of the contribution therefore remains unproven in the current manuscript.

major comments (3)
  1. [Section 5.3] The robustness experiment suffers from the same circularity. ChatGPT is prompted to paraphrase each level-1 code description while avoiding words from the original description, so the generated variations still start from the target text and share its semantic and lexical structure. This does not model the vocabulary shift, abbreviations, misspellings, or multi-condition phrasing typical of real clinician queries. The robustness claim would require testing on naturally occurring query variations, for example from search logs or elicited from domain experts, rather than only on paraphrases of the target descriptions.
  2. [Section 6] The conclusion states that SearchAI 'outperforms the manual search approach in terms of accuracy, robustness, performance, and scalability,' but no manual search baseline is reported anywhere in Section 5. The experiments compare only the authors' own algorithmic variants (default, flattened, hybrid). Without a manual-coding or existing-tool comparison, this claim is unsupported. At minimum, the authors should define a manual search protocol, measure its accuracy and latency on the same query set, and report inter-rater agreement among human coders.
  3. [Sections 4.1 and 5.2] The evaluation omits several technical details needed to interpret the headline accuracy. The similarity threshold in the default/hybrid predictors and the weights for vague terms are free parameters, yet the paper does not state how they were set (e.g., by tuning on a validation set) or report sensitivity analysis. Also, the reported 98%+ figures appear to be for level-1 codes only (Figures 3a/3b), while Section 6 claims 'more than 98% accurate' without qualification. Please report accuracy per hierarchy level, provide overall accuracy over all returned codes, and include confidence intervals or significance tests for the differences between variants.
minor comments (6)
  1. [Table 1] Typographical errors in code ranges: '100-199' should be 'I00-I99' and '000-09A' should be 'O00-O9A'.
  2. [Section 4.1] Step 4 of the pseudocode refers to a 'defined threshold' for approximate matching, but the threshold is never defined; please specify how it is chosen.
  3. [Section 5.4] The latency metric is ambiguous: it is unclear whether Figures 5a/5b report average per-query latency or total time for all level-1 queries, and the axes are not labeled.
  4. [Section 5.5] The text says 'Figures 6a show' (grammar), and the FFS sample sizes (200, 400, 840) do not match the stated fractions 25%, 50%, and 100%; please clarify the dataset construction.
  5. [Abstract and Section 4] The paper emphasizes 'generative AI,' but the hierarchical predictor described in Section 4 uses token-overlap matching and Levenshtein distance, with no generative component; this mismatch between the framing and the actual method should be addressed.
  6. [References] Reference [10] is cited for statistics about manual coding costs and ICD-9/ICD-10 comparisons, but the cited URL is a paper abstract page; please verify that the source supports these claims and provide a proper citation.

Circularity Check

2 steps flagged · score 7.0 of 10

Accuracy evaluation is circular: queries are generated from the ICD-10 descriptions stored in the search index, so the 98% figure largely measures near-duplicate retrieval rather than realistic clinical search.

  1. self definitional [Section 5.2 (Accuracy), page 8]
    "To do this, we convert each ICD code description into a corresponding question and assess how well the model interprets the information and retrieves all relevant ICD codes."

    The test query is generated from the ICD-10 description that the system is then asked to retrieve, and that same description is stored verbatim in the retrieval index. Section 4.1's default predictor first uses exact token overlap between query tokens and description tokens, so a query derived from a description is expected to match that description and lexically similar descendants. The reported accuracies of 98-99% therefore reflect text-to-self or near-duplicate matching, not the ability to map independently phrased clinical questions to codes. No held-out set of realistic clinician queries with independent gold labels is used.

  2. self definitional [Section 5.3 (Robustness), page 8]
    "To test robustness, we rephrased each code description in level-1 of both datasets using ChatGPT to produce semantically similar variants. We use the following prompt in ChatGPT to generate these variants of the original query: 'For each test case, avoid using words from the original description and generate a semantically equivalent search query that asks how a physician would record the disease or health condition.'"

    The robustness queries are also generated from the target ICD-10 descriptions that form the search index. Although the prompt discourages exact vocabulary reuse, the starting point is the answer text itself, so the paraphrases preserve the semantic content and structure of the stored description. This measures robustness to ChatGPT paraphrases of the stored text, not robustness to the vocabulary shift, abbreviations, misspellings, or multi-condition phrasing of real clinical queries. The robustness percentages therefore do not support the conclusion's general robustness claim.

full rationale

SearchAI's algorithmic components (hierarchy flattening, hybrid search, latency and scalability measurements) are described in a self-contained way and are not themselves derived from the accuracy claim. However, the central accuracy and robustness claims rest on test sets constructed from the ICD-10 descriptions that make up the retrieval index. Section 5.2 converts each code description into a question whose expected answer is that same description; with exact-token matching as the default predictor's first step, high accuracy is substantially a tautological result of query-to-stored-text similarity. Section 5.3 paraphrases those same descriptions with ChatGPT but still starts from the target string, so it does not introduce realistic clinical query variation. The paper reports no held-out realistic queries with independent gold labels, no precision/recall breakdown, no inter-rater agreement, and no comparison against manual search, even though the conclusion states that SearchAI is 'more than 98% accurate and outperforms the manual search approach.' The reported evidence therefore does not establish the central claim against a non-circular benchmark; the evaluation is partially circular by construction, while the system's engineering contributions remain independently describable.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central claim depends on two hand-tuned matching parameters and three unstated assumptions about the ICD hierarchy and the validity of the self-derived test set. No new medical entities or theoretical constructs are introduced.

free parameters (2)
  • similarity threshold = Unspecified
    The pseudocode in Sections 4.1 and 4.3 requires a predefined threshold for Levenshtein and embedding similarity matches, but the value is never stated. It is a hand-chosen parameter that directly controls which codes are returned.
  • vague term weights = Unspecified
    Section 4.1 mentions assigning lower weights to ambiguous terms like 'unspecified' and 'other', but no weighting scheme or values are given. This is an ad hoc adjustment introduced to improve accuracy.
assumptions (3)
  • domain assumption Token overlap and Levenshtein distance are sufficient proxies for semantic relevance between a query and a code description.
    The entire matching strategy in Section 4 assumes that clinical queries can be matched to ICD descriptions primarily by lexical similarity, with embeddings as a secondary signal. This may miss synonymous clinical terminology.
  • ad hoc to paper The ICD-10 hierarchy can be flattened when a child description has no token overlap with its parent, without losing valid code associations.
    Section 4.2 introduces hierarchy flattening based solely on token overlap. This assumes that non-overlapping descriptions are incorrectly placed in the hierarchy, which is not established by clinical coding rules.
  • ad hoc to paper Test queries generated by paraphrasing ICD code descriptions are representative of real clinical search queries.
    Section 5.2 constructs the test set by converting each code description into a question. This assumes that such self-derived queries capture the distribution of real user queries, which is questionable and unvalidated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Searching Clinical Data Using Generative AI." pith.science (2026). https://pith.science/paper/227WR3VO

@misc{pith2026250524090,
  author       = {Pith},
  title        = {Pith review of: Searching Clinical Data Using Generative AI},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/227WR3VO}},
  note         = {Machine review of arXiv:2505.24090}
}
read the original abstract

Artificial Intelligence (AI) is making a major impact on healthcare, particularly through its application in natural language processing (NLP) and predictive analytics. The healthcare sector has increasingly adopted AI for tasks such as clinical data analysis and medical code assignment. However, searching for clinical information in large and often unorganized datasets remains a manual and error-prone process. Assisting this process with automations can help physicians improve their operational productivity significantly. In this paper, we present a generative AI approach, coined SearchAI, to enhance the accuracy and efficiency of searching clinical data. Unlike traditional code assignment, which is a one-to-one problem, clinical data search is a one-to-many problem, i.e., a given search query can map to a family of codes. Healthcare professionals typically search for groups of related diseases, drugs, or conditions that map to many codes, and therefore, they need search tools that can handle keyword synonyms, semantic variants, and broad open-ended queries. SearchAI employs a hierarchical model that respects the coding hierarchy and improves the traversal of relationships from parent to child nodes. SearchAI navigates these hierarchies predictively and ensures that all paths are reachable without losing any relevant nodes. To evaluate the effectiveness of SearchAI, we conducted a series of experiments using both public and production datasets. Our results show that SearchAI outperforms default hierarchical traversals across several metrics, including accuracy, robustness, performance, and scalability. SearchAI can help make clinical data more accessible, leading to streamlined workflows, reduced administrative burden, and enhanced coding and diagnostic accuracy.

Figures

Figures reproduced from arXiv: 2505.24090 by the authors.

Figure 1
Figure 1. Hierarchical structure of ICD-10 code R07 (Pain in throat and chest). [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. The model architecture of SearchAI. disparate set of ICD codes. Therefore, we need better strategies to handle vague terms without compromising the overall accuracy of the code search process. Finally, flattening the hierarchy could be one way to simplify the search process. However, this is very risky: while its likely to improve accuracy at deeper levels (e.g., levels 2 and 3), the accuracies closer to the root le… view at source ↗
Figure 3
Figure 3. SearchAI accuracy on FFS and production datasets. [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: SearchAI accuracy with query variations on FFS and production datasets. [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Average latency on FFS and production datasets. [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Scalability using default and hybrid variant on FFS dataset. [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]
Figure 7
Figure 7. Figure 7: Scalability using default and hybrid variant on production dataset. [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

11 extracted references · 11 canonical work pages

  1. [1]

    AI in healthcare - statistics and facts

    Statista. AI in healthcare - statistics and facts. https://www.statista.com/topics/10011/ai-in-healthcare/ , 2024

    work page 2024

  2. [2]

    Diagnostic errors are common in seriously ill hospitalized adults

    San Francisco University of California. Diagnostic errors are common in seriously ill hospitalized adults. https://www.ucsf.edu/news/2024/01/426941/diagnostic-errors-are-common-seriously-ill\ -hospitalized-adults, 2024

    work page 2024

  3. [3]

    Auerbach, Tiffany M

    Andrew D. Auerbach, Tiffany M. Lee, Colin C. Hubbard, Sumant R. Ranji, Katie Raffel, Gilmer Valdes, John Boscardin, Anuj K. Dalal, Alyssa Harris, Ellen Flynn, Jeffrey L. Schnipper, and UPSIDE Research Group. Diagnostic errors in hospitalized adults who died or were transferred to intensive care. JAMA Internal Medicine, 184(2):164–173, 02 2024

    work page 2024

  4. [4]

    AI healthcare benefits

    IBM. AI healthcare benefits. https://www.ibm.com/think/insights/ai-healthcare-benefits , 2023

    work page 2023

  5. [5]

    ICD Code Lists

    CMS.gov. ICD Code Lists. https://www.cms.gov/medicare/coordination-benefits-recovery/ overview/icd-code-lists, 2024

    work page 2024

  6. [6]

    LOINC. LOINC. https://loinc.org/, 2025

    work page 2025

  7. [7]

    List of CPT/HCPCS Codes

    CMS.gov. List of CPT/HCPCS Codes. https://www.cms.gov/medicare/regulations-guidance/ physician-self-referral/list-cpt-hcpcs-codes , 2024

    work page 2024

  8. [8]

    Comparing Spark NLP for Healthcare and ChatGPT in Extracting ICD10-CM Codes from Clinical Notes

    Veysel Kocaman. Comparing Spark NLP for Healthcare and ChatGPT in Extracting ICD10-CM Codes from Clinical Notes. https://www.johnsnowlabs.com/comparing-spark-nlp-for-healthcare-and-chatgpt-in-extrac\ ting-icd10-cm-codes-from-clinical-notes/ , 2023

    work page 2023

Show all 11 references
  1. [9]

    ASUS. Miraico. https://aics.asus.com/miraico-en/, 2025

    2025

  2. [10]

    Few-shot ICD coding with knowledge transfer and evidence representation

    Fei Teng. Few-shot ICD coding with knowledge transfer and evidence representation. https://www.sciencedirect.com/science/article/abs/pii/S0957417423023631/, 2024

    2024

  3. [11]

    Akhloufi

    Moulay A. Akhloufi. Arxiv & prime ai style template. https://www.overleaf.com/latex/templates/ arxiv-and-prime-ai-style-template/qdnhqytdqzsc . Licensed under CC BY 4.0. 11

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.