Pith. sign in

REVIEW 46 references

Initial Investigation of LLM-Assisted Development of Rule-Based Clinical NLP System

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2506.16628 v1 pith:FY3TMW3G submitted 2025-06-19 cs.CL cs.LG

Jianlin Shi , Brian T. Bucher This is my paper
classification cs.CLcs.LG
keywords rule-baseddevelopmentclinicalsnippetssystemsexperimentsinitiallanguage
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Despite advances in machine learning (ML) and large language models (LLMs), rule-based natural language processing (NLP) systems remain active in clinical settings due to their interpretability and operational efficiency. However, their manual development and maintenance are labor-intensive, particularly in tasks with large linguistic variability. To overcome these limitations, we proposed a novel approach employing LLMs solely during the rule-based systems development phase. We conducted the initial experiments focusing on the first two steps of developing a rule-based NLP pipeline: find relevant snippets from the clinical note; extract informative keywords from the snippets for the rule-based named entity recognition (NER) component. Our experiments demonstrated exceptional recall in identifying clinically relevant text snippets (Deepseek: 0.98, Qwen: 0.99) and 1.0 in extracting key terms for NER. This study sheds light on a promising new direction for NLP development, enabling semi-automated or automated development of rule-based systems with significantly faster, more cost-effective, and transparent execution compared with deep learning model-based solutions.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

46 extracted references · 21 canonical work pages

  1. [1]

    & Reiss, F

    Chiticariu, L., Li, Y. & Reiss, F. R. Rule-Based Information Extraction is Dead! Long Live Rule-Based Information Extraction Systems! in Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing (eds. Yarowsky, D., Baldwin, T., Korhonen, A., Livescu, K. & Bethard, S.) 827–832 (Association for Computational Linguistics, Seattle...

    2013

  2. [2]

    Chiang, C.-C. et al. A large language model–based generative natural language processing framework fine-tuned on clinical notes accurately extracts headache frequency from electronic health records. Headache J. Head Face Pain 64, 400–409 (2024)

    2024

  3. [3]

    Wornow, M. et al. Zero-Shot Clinical Trial Patient Matching with LLMs. NEJM AI 2, AIcs2400360 (2025)

    2025

  4. [4]

    Peng, C. et al. Model tuning or prompt Tuning? a study of large language models for clinical concept and relation extraction. J. Biomed. Inform. 153, 104630 (2024)

    2024

  5. [5]

    & Wang, Y

    Sivarajkumar, S., Kelley, M., Samolyk-Mazzanti, A., Visweswaran, S. & Wang, Y. An Empirical Evaluation of Prompting Strategies for Large Language Models in Zero-Shot Clinical Natural Language Processing: Algorithm Development and Validation Study. JMIR Med. Inform. 12, e55318 (2024)

    2024

  6. [6]

    P., Virvou, M

    Panagoulias, D. P., Virvou, M. & Tsihrintzis, G. A. Augmenting Large Language Models with Rules for Enhanced Domain-Specific Interactions: The Case of Medical Diagnosis. Electronics 13, 320 (2024)

    2024

  7. [7]

    Wang, X. et al. RuleAlign: Making Large Language Models Better Physicians with Diagnostic Rule Alignment. Preprint at https://doi.org/10.48550/arXiv.2408.12579 (2024)

    work page Pith review arXiv doi:10.48550/arxiv.2408.12579 2024

  8. [8]

    & Kierner, Z

    Kierner, S., Kucharski, J. & Kierner, Z. Taxonomy of hybrid architectures involving rule-based reasoning and machine learning in clinical decision systems: A scoping review. J. Biomed. Inform. 144, 104428 (2023)

    2023

Show all 46 references
  1. [9]

    & Beam, A

    Kompa, B., Snoek, J. & Beam, A. L. Second opinion needed: communicating uncertainty in medical machine learning. Npj Digit. Med. 4, 1–6 (2021)

    2021

  2. [10]

    Cemri, M. et al. Why Do Multi-Agent LLM Systems Fail? Preprint at https://doi.org/10.48550/arXiv.2503.13657 (2025)

    2025 doi arXiv

  3. [11]

    Bucher, B. T. et al. Portable Automated Surveillance of Surgical Site Infections Using Natural Language Processing: Development and Validation. Ann. Surg. 272, 629 (2020)

    2020

  4. [12]

    Shi, J. et al. Identifying Patients Who Meet Criteria for Genetic Testing of Hereditary Cancers Based on Structured and Unstructured Family Health History Data in the Electronic Health Record: Natural Language Processing Approach. JMIR Med. Inform. 10, e37842 (2022)

    2022

  5. [13]

    Shi, J. et al. Natural language processing for the surveillance of postoperative venous thromboembolism. Surgery 170, 1175–1182 (2021)

    2021

  6. [14]

    & Hurdle, J

    Shi, J. & Hurdle, J. F. Trie-based rule processing for clinical NLP: A use-case study of n-trie, making the ConText algorithm more efficient and scalable. J. Biomed. Inform. 85, 106–113 (2018)

    2018

  7. [15]

    An efficient pipeline for high-performance information extraction of support clinical natural language processing

    Shi, J. An efficient pipeline for high-performance information extraction of support clinical natural language processing. (University of Utah, 2019)

    2019

  8. [16]

    Eyre, H. et al. Launching into clinical space with medspaCy: a new clinical text processing toolkit in Python. AMIA. Annu. Symp. Proc. 2021, 438–447 (2022)

    2022

  9. [17]

    Wei, J. et al. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. Preprint at https://doi.org/10.48550/arXiv.2201.11903 (2023)

    2023 doi arXiv

  10. [18]

    & Chen, C

    Jiang, R., Liu, L. & Chen, C. MoPE: Mixture of Prompt Experts for Parameter-Efficient and Scalable Multimodal Fusion. Preprint at https://doi.org/10.48550/arXiv.2403.10568 (2025). Appendix Note: the few shot examples are omitted here because of page limit. Snippet identificati...

    2025 doi arXiv

  11. [19]

    Is it valid or not?

    Verify the opinion expressed by another surgeon. Is it valid or not?

  12. [20]

    Do you agree with the other surgeon's opinion? Why?

  13. [21]

    Summarize your own opinion which will be the final decision

  14. [22]

    conclusion

    Generate a json-format output: either {{"conclusion":"yes"}} or {{"conclusion":"no"}} Keywords extraction prompts: Reasoning prompt: {SSI annotation guideline} -------------------------------- # Your Role: Clinical Informatist # Task: Identify the keywords from the given snipp...

  15. [23]

    Identify all anatomic sites or body parts or mentions indicating anatomic sites that potentially can be a surgical site (can potentially be used to link a procedure or infection mentioned outside the given snippet)

  16. [24]

    Identify all surgical or invasive procedures (can potentially be used to link an infection mentioned outside the given snippet)

  17. [25]

    Identify all infection or infection treatment

  18. [26]

    Identify all wound care

  19. [27]

    Identify all other infections that can be used to explain signs, symptoms, or lab abnormalities that might or might not be caused by SSI

  20. [28]

    Also, you need to double-check to make sure they keywords themselves are not over- generalized to get massive matches in other notes

    Make sure the keywords can actually be found in the original text and capture the key information that can be used for downstream analysis. Also, you need to double-check to make sure they keywords themselves are not over- generalized to get massive matches in other notes

  21. [30]

    'concerning for,' as these modifiers will be identified in downstream NLP components

    Prune the keywords to remove certainty, temporality, and experiencer modifiers, e.g. 'concerning for,' as these modifiers will be identified in downstream NLP components

  22. [31]

    concepts

    Summarize your findings in a json dictonary using template: {{"concepts": [], "expanded_concepts":[]}} -------------------------------- # For example: ### Given the input snippet: She underwent CT scan of the abdomen that showed an appendix dilated to 11 mm with adjacent fat s...

  23. [32]

    **Anatomic Sites or Body Parts (Surgical Sites):** - **Appendix:** This is a clear anatomic site that could be a surgical site in the context of appendectomy

  24. [33]

    It is more relevant for identifying anatomical findings and conditions rather than surgical interventions

    **Surgical or Invasive Procedures:** - **CT scan:** This is an imaging procedure, but it is not an invasive procedure. It is more relevant for identifying anatomical findings and conditions rather than surgical interventions

  25. [34]

    However, it is not specific to surgical site infections (SSIs)

    **Infection or Infection Treatment:** - **Inflammation:** This term indicates the presence of an inflammatory response, which could be related to an infection. However, it is not specific to surgical site infections (SSIs)

  26. [35]

    **Wound Care:** - **Not mentioned:** There is no mention of wound care in the snippet

  27. [36]

    It could be related to a non- surgical infection, such as appendicitis

    **Other Infections:** - **Inflammation:** This can be a sign of other infections, not specifically SSI. It could be related to a non- surgical infection, such as appendicitis

  28. [37]

    They are specific enough to capture relevant information without being overly generalized

    **Keywords in Original Text:** - **Appendix, inflammation, CT scan:** These are the key terms that can be used for downstream analysis. They are specific enough to capture relevant information without being overly generalized

  29. [38]

    **Expanded Concepts with Synonyms:** - **Appendix:** Appendix, cecum, right lower quadrant - **Inflammation:** Inflammation, inflammatory response - **CT scan:** CT scan, computed tomography

  30. [39]

    concepts

    **Pruned Keywords:** - Remove any certainty, temporality, or experiencer modifiers. The terms are already quite specific. ### Summary in JSON Dictionary: ```json {{ "concepts": ["Appendix", "Inflammation", "CT scan"], "expanded_concepts": ["Appendix", "Cecum", "Right lower qua...

  31. [40]

    Also, you need to double check to make sure the keywords themselves are not over generalized to get massive matches in other notes

    Identify the keywords directly from the snippet, make sure the keywords can actually found in original text and capture the key information that can be used to determine if a patient has SSI, even if the snippet doesn't have complete information. Also, you need to double check...

  32. [41]

    Based on your clinical knowledge, expand the keywords with synonyms, so that same meaning in a different snippet can also be found

  33. [42]

    'concerning for', as these modifiers will be identified in downstream NLP components

    Prune the keywords to remove certainty, temporality, and experiencer modifiers, e.g. 'concerning for', as these modifiers will be identified in downstream NLP components

  34. [43]

    Elaborate with detailed justification

  35. [44]

    # Your task: You will be given the clinical note snippet and the analysis from anther clinical informatist:

    Summarize your findings in a json dictionary. # Your task: You will be given the clinical note snippet and the analysis from anther clinical informatist:

  36. [45]

    Pay attention to the expanded keywords, make sure none of them is too broad

    Verify the analysis to check if it follows the above instructions step by step. Pay attention to the expanded keywords, make sure none of them is too broad

  37. [46]

    Double check the derived keywords list in json format

  38. [47]

    concepts

    Finalize the keywords in json format using the same structure: {{"concepts": [], "expanded_concepts":[]}} {text}

Pith tools