Pith. sign in

REVIEW 4 major objections 5 minor 45 references

Comprehensive Modeling and Question Answering of Cancer Clinical Practice Guidelines using LLMs

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A typed guideline graph lets an LLM answer cancer questions from guideline text rather than from memory.

desk verdict A useful systems description undermined by a circular few-shot evaluation and no direct check of answer correctness; the right call is major revision, not outright rejection without review. read the letter →

arxiv 2501.13984 v1 pith:DWTHGKBX submitted 2025-01-23 cs.CL cs.AIcs.LG

Bhumika Gupta , Pralaypati Ta , Keerthi Ram , Mohanasankar Sivaprakasam This is my paper · ORCID
classification cs.CLcs.AIcs.LG
keywords clinicalpracticeguidelinesknowledgegraphlargelanguagemodelsquestionansweringnodeclassificationfew-shotlearningNCCNnon-smallcelllungcancer
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that the treatment flows in an NCCN cancer guideline can be converted into a typed, context-enriched graph, and that this graph can ground a question-answering system where a large language model only writes the database query rather than the medical answer. The answer sentence is assembled from the retrieved subgraph's nodes and edges using a hand-built mapping, so the model never has to recall clinical facts from memory. The payoff, if the claim holds, is a guideline QA system that needs only 13 labeled query examples instead of the roughly 5,810 used by the prior deep-learning baseline, while keeping answers traceable to guideline text. The paper reports 80.86% zero-shot and 88.47% few-shot accuracy for automated node labeling, and 66.1% of auto-generated Cypher queries ran without error.

What carries the argument

The central object is the enriched guideline knowledge graph: 538 nodes typed as Disease Condition, Treatment Option, or Evaluation, carrying the guideline text plus a context property taken from page-level labels, connected by 707 directed edges typed as requires, indicates, or is followed by. The argument runs on three coupled mechanisms: LLM prompting for node labeling, LLM-based Cypher generation against the Neo4j graph, and a graph-semantics-to-text template (Table II) that converts each typed edge in the retrieved subgraph into a fixed natural-language sentence. The template step is what turns graph structure into clinically worded output.

What would settle it

Have independent oncologists check the answers generated for the paper's 59 held-out questions against the NCCN text, separating retrieval failures from wording failures: if a large fraction of correctly retrieved subgraphs still produce clinically wrong sentences because of the template, the factual-accuracy claim fails.

Watch

Extended reading notes

Core claim

The central discovery claimed is that a semantically enriched guideline graph is enough to make LLM-based question answering faithful: the LLM is confined to translating a natural-language question into Cypher, the graph database returns a subgraph, and a template mapping from relationship types to natural-language phrases turns each path into a clinical sentence such as "If the disease condition is X, use the treatment Y." The authors state that this design mitigates the hallucination risk that has blocked LLMs in medicine, because every generated statement corresponds to a node or edge present in the guideline graph. They support the claim with the node-classification accuracies, the error analysis of auto-generated Cypher queries, and the comparison with the prior deep-learning system that required far more training data.

Load-bearing premise

The load-bearing premise is that the hand-built mapping from graph relationships to natural-language sentences preserves the medical meaning of every pathway, since the paper never checks that mapping against expert clinician phrasing or a gold answer set.

Editorial extensions

If this is right

  • A guideline QA system can be built with 13 labeled query examples rather than thousands of question-answer pairs, which lowers the cost of updating answers when guidelines change.
  • Because every answer sentence is derived from an explicit graph path, a clinician can trace a recommendation back to the guideline nodes that generated it.
  • Few-shot node labeling at 88.47% makes it practical to enrich new guideline graphs automatically, with only a small set of corrected examples per guideline.
  • LLM failure modes appear before answer generation: roughly one third of Cypher queries need manual fixing, and content mismatches are the most common error.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A fair test of the approach should validate the Table II template mapping against oncologist-authored answers; the paper does not report such a validation, and a template that misphrases a relationship could produce fluent but misleading answers from a correctly retrieved subgraph.
  • The node-classification accuracy figures are computed against the authors' own manual labels, which is a reasonable first check but not the same as agreement with an independent clinician-annotated gold standard.
  • The same retrieve-then-render design could be applied to open-weight LLMs and to other guideline publishers, and the comparison the authors list as future work would show whether the Cypher-generation step depends on a particular model.
  • A direct ablation would compare the template-based answers with answers that quote the retrieved subgraph's node texts verbatim, isolating how much medical wording the templates contribute and whether they ever distort it.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper presents a method for converting the graphical flows of the NCCN Non-Small Cell Lung Cancer guideline into a knowledge graph with three node types and three relation types, using manual labeling and an automated LLM-based node classification with zero-shot and few-shot prompting. It then proposes a question-answering pipeline in which an LLM converts natural language questions into Cypher queries over a Neo4j database, retrieves a subgraph, and renders the subgraph as a natural language answer using a fixed template mapping. The authors report node classification accuracies of 80.86% (zero-shot) and 88.47% (few-shot), and report that 66.10% of automatically generated Cypher queries had no errors. They claim that the approach provides factually accurate answers and mitigates LLM hallucination while requiring far less training data than prior deep learning methods.

Significance. If the results were valid, the paper would offer a low-data method for building executable representations of cancer treatment guidelines and answering guideline-restricted questions. The paper has some strengths: it uses real NCCN guideline data, provides a clear error taxonomy for Cypher queries, gives full prompts for node classification, and makes the comparison between zero-shot and few-shot explicit. However, the two central claims are not supported. The few-shot node classification accuracy is obtained with few-shot examples selected from the test set, so the reported gain is not evidence of generalization. More importantly, the claimed 'factual accuracy' of the QA system is never evaluated; only Cypher query errors are measured, and the hand-crafted template mapping that produces the final answers is unvalidated. These issues are load-bearing rather than cosmetic.

major comments (4)
  1. [Section II.B] The few-shot node-classification evaluation is circular. The paper states that 'Twenty-three distinct nodes from the incorrect node classifications obtained using the previous technique are given as training instances' and then reports few-shot accuracy of 88.47% on the same set of nodes used for the zero-shot evaluation. Because the prompt examples are drawn from the very test instances being classified, the few-shot result is at least partly a fit to the test distribution and does not measure generalization. A valid few-shot evaluation would select training examples from a separate development split, not from the errors on the test set, or would use cross-validation.
  2. [Section IV.B and Abstract] The central claim that the method ensures 'factual accuracy' and mitigates hallucination is not evaluated anywhere. Section IV.B only reports error rates of the auto-generated Cypher queries (Table III), not the correctness of the final natural language answers. No reference answer set, no clinician review, and no automatic metric are applied to the generated answers. The two sample answers in the text are illustrative only and are not scored. Without an evaluation of answer correctness, the paper provides no evidence for its headline claim.
  3. [Section II.C and Table II] The hand-crafted graph-to-text mapping in Table II is unvalidated and may not preserve clinically important qualifiers. For instance, the template for 'Disease Condition requires Treatment Option' renders 'use the treatment <destination>' regardless of whether the node text says 'consider', 'preferred', 'category 2B', or includes conditions such as 'if not previously done'. Likewise, 'Treatment Option is followed by Treatment Option' becomes 'After the previous treatment is over, further use the treatment', dropping branching conditions like 'if progression'. Because the final answers are generated by these templates, even a correct subgraph extraction can produce a misleading or overcommittal answer, which directly undermines the claimed factual accuracy. The authors should validate the templates against expert paraphrases or a reference answer set.
  4. [Section IV.B] The comparison with prior work [15] is not well posed. The paper states that the proposed method requires only 13 training instances versus 5810 in [15], but it does not compare end-to-end answer accuracy. The only comparable quantity reported is Cypher query generation accuracy (66.1% no-error), which is not an answer accuracy. Thus the claim that the method achieves comparable or better QA performance with much less training data is unsupported.
minor comments (5)
  1. [Throughout] The manuscript uses 'NSCL' instead of the standard abbreviation 'NSCLC' for non-small cell lung cancer in several places, including the Abstract and Section II.A.
  2. [Section II.A] The manual node and relation labels serve as ground truth for the accuracy numbers, but no inter-annotator agreement or external validation is reported, so the reliability of the ground truth is unclear.
  3. [Table III] The 'Overall Error (%)' column appears to be computed over the combined test sets (e.g., 4/59 = 6.78% for Type-I), but this is not stated explicitly; the table should clarify the denominators used.
  4. [Section IV.B] The sample answer for the Set B question concatenates 'Stage IIIB (T4, N2) Stage IIIC (T4, N3)' without a conjunction, which is a readability issue and also illustrates how the template mapping flattens node boundaries.
  5. [References] Reference [40] points to a ChatGPT product page rather than to the official model documentation or paper for gpt-3.5-turbo-instruct; the citation should be updated.

Circularity Check

1 steps flagged · score 6.0 of 10

Few-shot node-labeling accuracy is partly circular: the 23 training examples are the zero-shot misclassifications on the full evaluation set, and accuracy is then reported on the same set.

  1. fitted input called prediction [Section II.B, 'LLM-powered Knowledge Model Enrichment' (few-shot example selection); Section IV.A, 'LLM-powered Node Classification' (reported accuracy)]
    "Twenty-three distinct nodes from the incorrect node classifications obtained using the previous technique are given as training instances with their correct manually annotated labels to boost classification accuracy using the few-shot learning paradigm."

    The few-shot 'training instances' are selected from the zero-shot misclassifications on the full node set, and no held-out split is described. The paper reports the few-shot accuracy on the same node set ('Upon using the few-shot learning of LLM, the accuracy was boosted to 88.47%.'), even though the correct manual labels of those 23 nodes are literally included in the prompt as examples. On a 538-node graph, 23 nodes correspond to about 4.3 percentage points of the 7.6-point reported improvement (80.86% to 88.47%), so more than half of the claimed gain is memorization of test labels, not prediction. The evaluation therefore reduces, in part, to fitting the answer key into the prompt.

full rationale

The paper's pipeline has three main stages: graph extraction (using the authors' prior tool [14]), manual node/relation labeling followed by LLM-based node labeling, and a graph-database QA system with LLM-generated Cypher queries and template-based answer generation. The zero-shot node-labeling accuracy (80.86%) is a legitimate agreement measurement against the authors' manual labels, though those labels are not externally validated. The few-shot result (88.47%) is partially circular because the 23 in-context examples are exactly nodes that zero-shot misclassified on the evaluation set, and the same set is scored again; including their correct labels in the prompt memorizes those instances, making the reported improvement partly forced by construction. I did not count the QA query-generation evaluation as circular: its train and test question sets are disjoint, and the paper's admission that 'the model tends to follow the provided example queries' is a generalization limitation, not a reduction to the test set. I also did not count the self-citation to [14] for the extraction tool as load-bearing circularity, because the graph is manually reviewed for errors before enrichment and the node/relation labels are created afresh in this paper. Finally, the unvalidated Table II relation-to-template mapping and the absence of an end-to-end answer-accuracy evaluation are serious correctness and factual-accuracy risks, but they are missing validation rather than circular derivations. Overall, one central quantitative claim partially reduces to memorizing test labels, so the circularity score is 6.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical or conceptual entities. The free parameters are design choices in the experimental setup, most notably the few-shot example selections, which directly shape the reported accuracy numbers. The axioms are the unstated assumptions about the completeness and correctness of the manual labeling and the template mapping.

free parameters (3)
  • few-shot node classification examples (23 nodes) = 23 incorrectly classified nodes from zero-shot run
    The examples are hand-selected from the zero-shot errors on the test set. Their content directly influences the reported few-shot accuracy (88.47%), making the result dependent on this choice.
  • few-shot QA query examples (13 questions) = 3 from Set A + 10 from Set B
    The LLM is given these examples as in-context demonstrations for Cypher generation. The paper observes the model tends to follow the provided examples, so the selection of these 13 questions is a hand-chosen parameter affecting the query generation results.
  • path length limits in example Cypher queries = varying values such as [*2..5], [*1..4]
    The example queries use arbitrary maximum path lengths, and the paper reports 'connection length error' when the LLM chooses different lengths. These hand-set values influence which subgraph is retrieved and therefore the answer content.
assumptions (4)
  • domain assumption The three manually defined node classes (Disease Condition, Treatment Option, Evaluation) are sufficient to categorize every node in the guideline.
    Section II.A defines these classes based on a literature review, but there is no inter-annotator agreement or external validation that all 538 nodes fit neatly into one of the three categories.
  • domain assumption The three relation types (requires, indicates, is followed by) fully capture the clinical relationships expressed in the guideline flows.
    Section II.A and Figure 2 present this taxonomy as complete, but no justification is given that more nuanced relations (e.g., contraindicates, alternative to) are unnecessary.
  • domain assumption The graph extracted by the authors' prior tool [14] is a faithful representation of the paper guideline after manual review, with 'zero error'.
    Section II.A claims the graph is reviewed for false connections to ensure zero error, but no external audit or comparison is provided.
  • ad hoc to paper The natural language templates in Table II preserve the medical meaning of the graph relations when converted to text.
    These templates are invented by the authors and are not validated against clinician-authored answers or a reference answer set, yet they are the sole mechanism for generating final answers.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Comprehensive Modeling and Question Answering of Cancer Clinical Practice Guidelines using LLMs." pith.science (2026). https://pith.science/paper/DWTHGKBX

@misc{pith2026250113984,
  author       = {Pith},
  title        = {Pith review of: Comprehensive Modeling and Question Answering of Cancer Clinical Practice Guidelines using LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DWTHGKBX}},
  note         = {Machine review of arXiv:2501.13984}
}
read the original abstract

The updated recommendations on diagnostic procedures and treatment pathways for a medical condition are documented as graphical flows in Clinical Practice Guidelines (CPGs). For effective use of the CPGs in helping medical professionals in the treatment decision process, it is necessary to fully capture the guideline knowledge, particularly the contexts and their relationships in the graph. While several existing works have utilized these guidelines to create rule bases for Clinical Decision Support Systems, limited work has been done toward directly capturing the full medical knowledge contained in CPGs. This work proposes an approach to create a contextually enriched, faithful digital representation of National Comprehensive Cancer Network (NCCN) Cancer CPGs in the form of graphs using automated extraction and node & relationship classification. We also implement semantic enrichment of the model by using Large Language Models (LLMs) for node classification, achieving an accuracy of 80.86% and 88.47% with zero-shot learning and few-shot learning, respectively. Additionally, we introduce a methodology for answering natural language questions with constraints to guideline text by leveraging LLMs to extract the relevant subgraph from the guideline knowledge base. By generating natural language answers based on subgraph paths and semantic information, we mitigate the risk of incorrect answers and hallucination associated with LLMs, ensuring factual accuracy in medical domain Question Answering.

Figures

Figures reproduced from arXiv: 2501.13984 by the authors.

Figure 1
Figure 1. A section of the enriched guideline knowledge base of NCCN NSCL CPG. The figure on the left is a guideline page highlighting the node (in the [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Classification of relations between node types. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Node Labeling using LLM with zero-shot learning. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (2 more)
Figure 5
Figure 5. Figure 5: NCCN NSCL CPG Question-Answering system using LLM. [PITH_FULL_IMAGE:figures/full_fig_p004_5.png]
Figure 4
Figure 4. Figure 4: Node Labeling using LLM with few-shot learning. [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

45 extracted references · 39 canonical work pages

  1. [15]

    Decision Knowledge Graphs: Construction of and Usage in Question Answering for Clinical Practice Guidelines

    V . V . Kandula and P. Bhattacharyya, “Decision Knowledge Graphs: Construction of and Usage in Question Answering for Clinical Practice Guidelines,” Aug. 2023, arXiv:2308.02984 [cs]. [Online]. Available: http://arxiv.org/abs/2308.02984

    work page Pith review arXiv 2023

  2. [1]

    Treatment by cancer type,

    “Treatment by cancer type,” https://www.nccn.org/guidelines/category 1, (accessed Apr. 8, 2024)

    work page 2024

  3. [2]

    Guidelines, tools, & resources,

    “Guidelines, tools, & resources,” https://society.asco.org/practice- patients/guidelines, (accessed Apr. 8, 2024)

    work page 2024

  4. [3]

    4, 2024)

    ESMO, “ESMO,” https://www.esmo.org/, (accessed Apr. 4, 2024)

    work page 2024

  5. [4]

    AI Driven Knowledge Extraction from Clinical Practice Guidelines: Turning Research into Practice

    M. Hussain, J. Hussain, T. Ali, F. A. Satti, and S. Lee, “AI Driven Knowledge Extraction from Clinical Practice Guidelines: Turning Research into Practice,” Dec. 2020, arXiv:2012.05489 [cs]. [Online]. Available: http://arxiv.org/abs/2012.05489

    work page Pith review arXiv 2020

  6. [5]

    Comprehensive analysis of rule formalisms to represent clinical guidelines: Selection criteria and case study on antibiotic clinical guidelines,

    N. Iglesias, J. M. Juarez, and M. Campos, “Comprehensive analysis of rule formalisms to represent clinical guidelines: Selection criteria and case study on antibiotic clinical guidelines,” Artificial Intelligence in Medicine , vol. 103, p. 101741, Mar. 2020. [Online]. Available: https://linkinghub.elsevier.com/retrieve/pii/S0933365719301873

    work page 2020

  7. [6]

    GDSI: a Web-based decision support system to facilitate the efficient and effective use of clinical practice guidelines,

    D. Stahl, L. Rouse, D. Ko, and J. Niland, “GDSI: a Web-based decision support system to facilitate the efficient and effective use of clinical practice guidelines,” in 37th Annual Hawaii International Conference on System Sciences, 2004. Proceedings of the . Big Island, HI, USA: IEEE, 2004, p. 10 pp. [Online]. Available: http://ieeexplore.ieee.org/documen...

    work page arXiv 2004

  8. [7]

    A knowledge-based approach to assign breast cancer treatments in oncology units,

    T. G. Miranda-Mena, S. L. Ben ´ıtez U., J. L. Ochoa, R. Mart ´ınez-B´ejar, J. T. Fern ´andez-Breis, and J. Salinas, “A knowledge-based approach to assign breast cancer treatments in oncology units,” Expert Systems with Applications, vol. 31, no. 3, pp. 451–457, Oct. 2006. [Online]. Available: https://linkinghub.elsevier.com/retrieve/pii/S0957417405002599

    work page 2006

Show all 45 references
  1. [8]

    Design and Implementation of Medical Process Visualization CDSS Oriented to NCCN Guidelines,

    M. Liu, J. Zhang, and X. Lin, “Design and Implementation of Medical Process Visualization CDSS Oriented to NCCN Guidelines,” in Proceedings of the 2020 International Conference on Internet Computing for Science and Engineering . Male Maldives: ACM, Jan. 2020, pp. 30–34. [Onlin...

    2020

  2. [9]

    Translating an evidence-based clinical pathway into shareable CDS: developing a systematic process using publicly available tools,

    J. J. Michel, E. J. Flores, L. Dutcher, N. K. Mull, and A. Y . Tsou, “Translating an evidence-based clinical pathway into shareable CDS: developing a systematic process using publicly available tools,” Journal of the American Medical Informatics Association , vol. 28, no. 1, p...

    2021

  3. [10]

    Ontology Based Modeling and Execution of Nursing Care Plans and Practice Guidelines,

    M. A. Din, S. S. R. Abidi, and B. Jafarpour, “Ontology Based Modeling and Execution of Nursing Care Plans and Practice Guidelines,” pp. 1104–1108, 2010

    2010

  4. [11]

    An overview of clinical decision support systems: benefits, risks, and strategies for success,

    R. T. Sutton, D. Pincock, D. C. Baumgart, D. C. Sadowski, R. N. Fedorak, and K. I. Kroeker, “An overview of clinical decision support systems: benefits, risks, and strategies for success,”npj Digital Medicine, vol. 3, no. 1, pp. 1–10, Feb. 2020, publisher: Nature Publishing Gr...

    2020

  5. [12]

    Some Unintended Consequences of Clinical Decision Support Systems,

    J. S. Ash, D. F. Sittig, E. M. Campbell, K. P. Guappone, and R. H. Dykstra, “Some Unintended Consequences of Clinical Decision Support Systems,” AMIA Annual Symposium Proceedings , vol. 2007, p. 26, 2007, publisher: American Medical Informatics Association. [Online]. Available...

    2007

  6. [13]

    Rule-based clinical decision support system using the openehr standard,

    S. T. Silva, F. Hak, and J. Machado, “Rule-based clinical decision support system using the openehr standard,” Procedia Comput. Sci., vol. 201, pp. 726–731, 2022

    2022

  7. [14]

    Automated Knowledge Modeling for Cancer Clinical Practice Guidelines,

    P. Ta, B. Gupta, A. Jain, S. S. C, A. Sarkar, K. Ram, and M. Sivaprakasam, “Automated Knowledge Modeling for Cancer Clinical Practice Guidelines,” in 2023 45th Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC). Sydney, Australia: IEEE...

    2023

  8. [16]

    A Review on Knowledge Graphs for Healthcare: Resources, Applications, and Promises,

    H. Cui, J. Lu, S. Wang, R. Xu, W. Ma, S. Yu, Y . Yu, X. Kan, C. Ling, T. Fu, L. Zhao, J. Ho, F. Wang, and C. Yang, “A Review on Knowledge Graphs for Healthcare: Resources, Applications, and Promises,” Feb. 2024, arXiv:2306.04802 [cs]. [Online]. Available: http://arxiv.org/abs/...

    2024 arXiv

  9. [17]

    GPT-3.5, GPT-4, or BARD? Evaluating LLMs reasoning ability in zero-shot setting and performance boosting through prompts,

    J. L. Espejel, E. H. Ettifouri, M. S. Y . Alassan, E. M. Chouham, and W. Dahhane, “GPT-3.5, GPT-4, or BARD? Evaluating LLMs reasoning ability in zero-shot setting and performance boosting through prompts,” Natural Language Processing Journal , vol. 5, p. 100032, 2023. [Online]...

    2023

  10. [18]

    Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies,

    G. V . Aher, R. I. Arriaga, and A. T. Kalai, “Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies,” in Proceedings of the 40th International Conference on Machine Learning. PMLR, Jul. 2023, pp. 337–371, iSSN: 2640-3498. [Online]. Availab...

    2023

  11. [19]

    LLM Based Generation of Item-Description for Recommendation System,

    A. Acharya, B. Singh, and N. Onoe, “LLM Based Generation of Item-Description for Recommendation System,” in Proceedings of the 17th ACM Conference on Recommender Systems . Singapore Singapore: ACM, Sep. 2023, pp. 1204–1207. [Online]. Available: https://dl.acm.org/doi/10.1145/3...

    2023

  12. [20]

    A Survey of Large Language Models,

    W. X. Zhao, K. Zhou, J. Li, T. Tang, X. Wang, Y . Hou, Y . Min, B. Zhang, J. Zhang, Z. Dong, Y . Du, C. Yang, Y . Chen, Z. Chen, J. Jiang, R. Ren, Y . Li, X. Tang, Z. Liu, P. Liu, J.-Y . Nie, and J.-R. Wen, “A Survey of Large Language Models,” Nov. 2023, arXiv:2303.18223 [cs]....

    2023 arXiv

  13. [21]

    Open-Source Large Language Models Outperform Crowd Workers and Approach ChatGPT in Text- Annotation Tasks,

    M. Alizadeh, M. Kubli, Z. Samei, S. Dehghani, J. D. Bermeo, M. Korobeynikova, and F. Gilardi, “Open-Source Large Language Models Outperform Crowd Workers and Approach ChatGPT in Text- Annotation Tasks,” 2023, publisher: [object Object] Version Number:

    2023

  14. [22]

    Available: https://arxiv.org/abs/2307.02179

    [Online]. Available: https://arxiv.org/abs/2307.02179

    arXiv

  15. [23]

    Large language models for data annotation: A survey,

    Z. Tan, A. Beigi, S. Wang, R. Guo, A. Bhattacharjee, B. Jiang, M. Karami, J. Li, L. Cheng, and H. Liu, “Large language models for data annotation: A survey,” ArXiv, vol. abs/2402.13446, 2024. [Online]. Available: https://api.semanticscholar.org/CorpusID:267770019

    2024 arXiv

  16. [24]

    Multi-perspective improvement of knowledge graph completion with large language models,

    D. Xu, Z. Zhang, Z. Lin, X. Wu, Z. Zhu, T. Xu, X. Zhao, Y . Zheng, and E. Chen, “Multi-perspective improvement of knowledge graph completion with large language models,” 2024

    2024

  17. [25]

    Language models are few- shot learners,

    T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-V oss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B....

    2005 arXiv

  18. [26]

    Few- shot adaptation works with UnpredicTable data,

    J. S. Chan, M. Pieler, J. Jao, J. Scheurer, and E. Perez, “Few- shot adaptation works with UnpredicTable data,” in Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , A. Rogers, J. Boyd- Graber, and N. Okazaki, Eds....

    2023

  19. [27]

    Assessing the research landscape and clinical utility of large language models: a scoping review,

    Y .-J. Park, A. Pillai, J. Deng, E. Guo, M. Gupta, M. Paget, and C. Naugler, “Assessing the research landscape and clinical utility of large language models: a scoping review,” BMC Medical Informatics and Decision Making , vol. 24, no. 1, p. 72, Mar. 2024. [Online]. Available:...

    2024 doi

  20. [28]

    Uncertainty-Aware Text- to-Program for Question Answering on Structured Electronic Health Records,

    D. Kim, S. Bae, S. Kim, and E. Choi, “Uncertainty-Aware Text- to-Program for Question Answering on Structured Electronic Health Records,” in Proceedings of the Conference on Health, Inference, and Learning . PMLR, Apr. 2022, pp. 138–151, iSSN: 2640-3498. [Online]. Available: h...

    2022

  21. [29]

    Guidelines detail - non-small cell lung cancer,

    “Guidelines detail - non-small cell lung cancer,” https://www.nccn.org/guidelines/guidelines-detail?category=1id=1450, (accessed Apr. 8, 2024)

    2024

  22. [30]

    Snomed ct,

    “Snomed ct,” https://www.nlm.nih.gov/healthit/snomedct/index.html, 2016, (accessed Apr. 8, 2024)

    2016

  23. [31]

    NCI thesaurus,

    “NCI thesaurus,” https://ncithesaurus.nci.nih.gov/ncitbrowser/, (accessed Apr. 8, 2024)

    2024

  24. [32]

    MeSH - NCBI,

    “MeSH - NCBI,” https://www.ncbi.nlm.nih.gov/mesh/, (accessed Apr. 8, 2024)

    2024

  25. [33]

    The unified medical language system (umls): integrating biomedical terminology,

    O. Bodenreider, “The unified medical language system (umls): integrating biomedical terminology,” Nucleic acids research , vol. 32 Database issue, pp. D267–70, 2004. [Online]. Available: https://api.semanticscholar.org/CorpusID:205228801

    2004

  26. [34]

    Classification, Ontology, and Precision Medicine,

    M. A. Haendel, C. G. Chute, and P. N. Robinson, “Classification, Ontology, and Precision Medicine,” The New England journal of medicine, vol. 379, no. 15, pp. 1452–1462, Oct. 2018. [Online]. Available: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6503847/

    2018

  27. [35]

    Semantic Health Knowledge Graph: Semantic Integration of Heterogeneous Medical Knowledge and Services,

    L. Shi, S. Li, X. Yang, J. Qi, G. Pan, and B. Zhou, “Semantic Health Knowledge Graph: Semantic Integration of Heterogeneous Medical Knowledge and Services,” BioMed Research International, vol. 2017, p. 2858423, 2017. [Online]. Available: https://www.ncbi.nlm.nih.gov/pmc/articl...

    2017

  28. [36]

    Developing an Ontology for Encoding Disease Treatment Information in Medical Abstracts,

    C. S. G. Khoo, J.-C. Na, V . W. Wang, and S. Chan, “Developing an Ontology for Encoding Disease Treatment Information in Medical Abstracts,” DESIDOC Journal of Library & Information Technology, vol. 31, no. 2, Feb. 2011, number: 2. [Online]. Available: https://publications.drd...

    2011

  29. [37]

    Modeling of Clinical Practice Guidelines for an Interactive Decision Support Using Ontologies,

    P. Philipp, M. Bommersheim, S. Robert, D. Hempel, and J. Beyerer, “Modeling of Clinical Practice Guidelines for an Interactive Decision Support Using Ontologies,” LasVegas/Nev., 2018. [Online]. Available: https://publica.fraunhofer.de/handle/publica/407205

    2018

  30. [38]

    Hl7 fhir: An agile and restful approach to healthcare information exchange,

    D. Bender and K. Sartipi, “Hl7 fhir: An agile and restful approach to healthcare information exchange,” Proceedings of the 26th IEEE International Symposium on Computer-Based Medical Systems , pp. 326–331, 2013. [Online]. Available: https://api.semanticscholar.org/CorpusID:11069145

    2013

  31. [39]

    Neo4j cypher® manual,

    “Neo4j cypher® manual,” https://neo4j.com/docs/cypher- manual/current/introduction/cypher overview/, (accessed Apr. 8, 2024)

    2024

  32. [40]

    Query a neo4j database using cypher,

    “Query a neo4j database using cypher,” https://neo4j.com/docs/getting- started/cypher-intro/, (accessed Apr. 8, 2024)

    2024

  33. [41]

    ChatGPT - open a I gpt 3.5,

    “ChatGPT - open a I gpt 3.5,” https://chat.openai.com/g/g-F00faAwkE- open-a-i-gpt-3-5, model - (gpt-3.5-turbo-16k)(accessed Apr. 8, 2024)

    2024

  34. [42]

    Neo4j graph database & analytics,

    “Neo4j graph database & analytics,” https://neo4j.com/, (accessed Apr. 8, 2024)

    2024

  35. [43]

    Text2Cypher - natural language queries,

    “Text2Cypher - natural language queries,” https://neo4j.com/labs/neodash/2.4/user-guide/extensions/natural- language-queries/, (accessed Apr. 8, 2024)

    2024

  36. [44]

    NeoDash - Dashboard Builder for Neo4j,

    “NeoDash - Dashboard Builder for Neo4j,” https://neo4j.com/labs/neodash/, (accessed Apr. 8, 2024)

    2024

  37. [45]

    FlexKBQA: A Flexible LLM-Powered Framework for Few-Shot Knowledge Base Question Answering,

    Z. Li, S. Fan, Y . Gu, X. Li, Z. Duan, B. Dong, N. Liu, and J. Wang, “FlexKBQA: A Flexible LLM-Powered Framework for Few-Shot Knowledge Base Question Answering,” Proceedings of the AAAI Conference on Artificial Intelligence , vol. 38, no. 17, pp. 18 608–18 616, Mar. 2024. [Onl...

    2024

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.