Pith. sign in

REVIEW 4 major objections 4 minor 88 references

Way to Specialist: Closing Loop Between Specialized LLM and Evolving Domain Knowledge Graph

T0 review · 4 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A knowledge graph that grows as an LLM answers questions can specialize it for a domain, with no fine-tuning.

desk verdict Clever closed-loop RAG/KG system, but the main evaluation leaks test labels through the DKG, so the headline gains are unsupported. read the letter →

arxiv 2411.19064 v1 pith:4V7E3LKM submitted 2024-11-28 cs.CL cs.AI

Yutong Zhang , Lixing Chen , Shenghong Li , Nan Cao , Yang Shi , Jiaxin Ding , Zhe Qu , Pan Zhou
show 1 more author
Yang Bai
This is my paper · ORCID
classification cs.CLcs.AI MSC 68T5068T07
keywords specializedLLMsdomainknowledgegraphretrieval-augmentedgenerationevolutionLLM-KGfeedbackloopdomain-specificquestionansweringtriplevectordatabaseretrieval
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a general-purpose LLM can become a domain specialist without any fine-tuning, by building and continuously consulting its own domain knowledge graph as it answers questions. The Way-to-Specialist (WTS) framework closes a loop between two roles for the same LLM: a reasoner that retrieves and prunes knowledge triples to answer the current question, and a knowledge engineer that converts the question plus its gold answer into new triples that update the graph for the next question. Because the graph can start empty, the method targets domains where no curated knowledge graph exists. Across six QA datasets in five domains, the authors report that WTS surpasses the previous state of the art in four specialized domains, with the largest gain a relative 11.3% accuracy improvement on MedMCQA multiple-choice questions.

What carries the argument

The load-bearing mechanism is the LLM$\circlearrowright$KG feedback loop: bidirectional enhancement between the LLM and a vector-database-stored domain knowledge graph (DKG), a collection of (subject, relation, object) knowledge triples. In the DKG-Augmented LLM half, entity extraction, iterative depth-wise retrieval (exact match plus cosine-similarity filtering with gap threshold $L$), LLM semantic scoring with top-$K$ pruning, and an LLM confidence-based early exit decide what knowledge reaches the prompt. In the LLM-Assisted DKG Evolution half, a schema-free LLM prompt generates candidate triples from question, gold answer, and retrieved triples, and a redundancy check (exact existence plus similarity threshold $L'$) admits only new knowledge. The connector is the evolving DKG itself: it is empty at initialization, grows as questions are answered, and is the sole channel through which experience carries over.

What would settle it

Run WTS with the DKG grown only from a disjoint training split, never touching test gold answers, and compare against the sequential-apprenticeship numbers: if the 11.3% margin over the prior SOTA shrinks or disappears, the per-question gold answers entering the graph carry the reported gain. A second check: if accuracy over baseline rises steadily with question position in the test sequence, the graph is genuinely accumulating useful knowledge, while a flat profile would point to prompt-side effects.

Watch

Extended reading notes

Core claim

WTS's central claim is that specialization can be achieved as a closed loop rather than as a training procedure. In the LLM$\circlearrowright$KG paradigm, the same LLM plays both roles: as a reasoner it retrieves question-relevant triples from a domain knowledge graph (DKG), prunes them by semantic relevance, and reasons over them; as a knowledge engineer it generates new triples from the question, its retrieved context, and the gold answer, checks them for redundancy, and inserts them into the DKG. The loop closes because the DKG used to answer question $q$ is exactly the DKG that the previous questions built. The paper validates this on six QA datasets across five domains using GPT-3.5 and GPT-4o as backbones, reporting that WTS exceeds the previous best prompting-based method in all four specialized domains and reaches a maximum improvement of 11.3% over the current SOTA.

Load-bearing premise

The load-bearing premise is that feeding each test question's gold answer into the growing knowledge graph before the next test question is answered is a fair way to measure the method, rather than a quiet form of label leakage.

Editorial extensions

If this is right

  • WTS with GPT-4o beats standard prompting, chain-of-thought, and the graph-reasoning baseline Think-on-Graph on all four specialized domains tested, with the largest margins on medical QA.
  • Because the framework starts from an empty knowledge graph, it can be deployed in domains that have no pre-built KG; the DKG is constructed on the fly from questions and gold answers.
  • A stronger base model (GPT-4o vs GPT-3.5) retrieves less deeply and builds a more compact DKG while scoring higher, suggesting the loop partially compensates for weaker parametric knowledge.
  • Deeper retrieval improves accuracy up to a point (MedMCQA single-choice rises from 0.358 at depth 1 to 0.622 at depth 3, then declines), after which irrelevant triples hurt performance and raise cost.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct test of the loop would split each test set by position: if accuracy over baseline rises as the graph grows across the first, second, and third thirds of the sequence, the accumulated graph is the carrier; a flat profile would suggest the gains come mainly from the richer prompt format rather than the stored triples.
  • The apprenticeship protocol feeds every test question's gold answer into the graph before the next question, so the headline margin mixes learning with evaluation; reading the loop as a deployment would require building the graph only during a separate apprenticeship phase and freezing it for scoring, as the paper does only for ChatDoctor5k and not for the datasets behind the 11.3% figure.
  • Because LLM-generated triples are inserted after redundancy checks but no fact verification, a wrong triple can bias later retrievals; scoring each candidate triple's confidence before insertion would reveal how much of the gain depends on graph quality rather than graph quantity.
  • The Mastership stage replaces gold answers with user feedback, which suggests a product path: the same loop could adapt a general assistant to a specific user's vocabulary and needs over time, with the DKG acting as a personalized, growing memory.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes Way-to-Specialist (WTS), a retrieval-augmented generation framework that couples a DKG-augmented LLM with LLM-assisted DKG evolution. The DKG starts empty and is populated with triples generated from each answered question, and those triples are then used to answer later questions. The authors claim state-of-the-art performance on four specialized domains with up to 11.3% improvement on MedMCQA, based on experiments over six datasets spanning medical, natural science, social science, linguistics, and general QA.

Significance. The 'LLM-circle-arrow-KG' loop is conceptually appealing and, if validated, could reduce the need for domain-specific fine-tuning. The algorithm and prompt templates are described in enough detail to make a proper re-evaluation feasible. However, the main empirical evaluation suffers from label leakage because gold answers from the test set are used to evolve the DKG for subsequent test questions; the claimed SOTA gains are therefore not supported. The only clean split (Appendix Table 6) shows a much smaller gain on a single dataset. As such, the significance of the contribution is currently unsubstantiated.

major comments (4)
  1. [§3.2.2, §3.3, §4.1.2] The evaluation protocol leaks gold answers into the knowledge base. In §3.2.2, the domain knowledge generation step is defined as T+_q = LLMGen(q, α*_q, ...), where α*_q is the gold answer; §3.3 then updates the DKG to G_{q+1} using T+_q before question q+1 is answered. Since §4.1.2 states that WTS is initialized with an empty DKG for all datasets and the main experiments process the test set sequentially, the DKG used for later test questions contains triples derived from earlier test questions' gold answers. This constitutes label leakage and invalidates the headline results in Table 1, including the reported 11.3% relative improvement on MedMCQA. The only clean evaluation, Appendix Table 6, uses a separate 800/200 apprenticeship/mastership split on ChatDoctor5k and shows a BERTScore gain of 0.009 over the GPT-3.5 baseline (0.792 vs. 0.783), far smaller than the main reported gains. The main experiments must be re-run with an apprenticeship split (or a pre-built DKG) and then evaluated on held-out questions.
  2. [§4.2.1, Tables 3 and 5] The maximum retrieval depth D is selected per dataset on the test set. Table 3 reports accuracy for varying D on ChatDoctor5k, PubMedQA, and MedMCQA, and Table 5 lists the D values used to produce the main Table 1 results. Because D is chosen by comparing performance on the same data that is later reported, the main results are optimistically biased by test-set overfitting. A validation split or a single fixed D across all datasets is required to support the reported performance.
  3. [§4.2.2, Figure 7] Figure 7 presents the accuracy improvement over GPT-3.5 as the number of processed Q&A samples increases, along with DKG size, for the medical datasets. Since these curves are produced on the same test set whose gold answers are used to evolve the DKG, the upward trend is expected under the leakage protocol and does not provide evidence of genuine progressive specialization. The fluctuation explanation in the text ('uncertain knowledge overlap') does not address this confound.
  4. [§4.1.1 and Table 1] The 'previous SOTA' claim is not supported by the chosen baselines. The comparison includes only I/O prompting (GPT-3.5, GPT-4o), CoT, and ToG; no other KG-augmented RAG systems or domain-specialized LLMs (e.g., medical RAG methods such as MindMap or MedConQA) are included. Even setting aside the leakage issue, exceeding these three baselines does not establish that WTS surpasses the previous state of the art in four specialized domains.
minor comments (4)
  1. [Figures 4–8 and body text] Several figures and body text passages contain corrupted tokens beginning with '/uni00000026/...' (e.g., the axis labels in Figure 4 and the curve labels in Figure 7). These need to be repaired before publication.
  2. [Appendix B.1 and Table 1] The dataset is referred to as 'Simple Questions' in Appendix B.1 and as 'SimpleQA' in Table 1 and §4.2.1; please unify the naming.
  3. [§4.2.1] The statement that GPT-4o underperforms GPT-3.5 on ChatDoctor5k because of GPT-4o's 'conservation' is speculative and should be supported with evidence or removed.
  4. [Table 5] Table 5 reports for MedMCQA(Multi) an execution time of 17.17s at D3 versus 18.77s at D2, which is non-monotonic; please verify the values or explain the variance.

Circularity Check

1 steps flagged · score 8.0 of 10

Main-table gains are inflated by test-label leakage: DKG evolution uses gold answers, so later test questions are answered with triples built from earlier answer keys.

  1. fitted input called prediction [Section 3.2.2 (Domain Knowledge Generation), Section 3.3 (Formation Pipeline), Section 4.1.2 (Hyperparameters)]
    "Given question q and the gold answer α∗q, the module prompts LLM to create KG triples T+q = LLMGen(q, α∗q, ¯T(D)q ) ... The generated triples T+q are then incorporated to update DKG from Gq to Gq+1. ... For all datasets, WTS is initialized with an empty vector database as its DKG."

    In the main experiments, the sequence of test questions is the stream on which WTS "learns": the DKG starts empty, and for each question the gold answer α*_q is used to generate triples that are added to the DKG before the next question is answered. Because retrieved triples from this evolving DKG are inserted into the prompt for subsequent questions, the model's later answers are conditioned on the gold answers of earlier test items. The reported accuracy gain and the "continuous improvement" curves in Figure 7 therefore reflect, by construction, access to the answer key; they are not measurements on unseen data.

full rationale

The WTS architecture itself is internally well-defined: DKG-Augmented LLM and LLM-Assisted DKG Evolution are specified with explicit prompts and equations, and no load-bearing conclusion depends on a self-citation chain. The circularity is in the empirical evaluation. Section 3.2.2 defines triple generation with the gold answer, Section 3.3 defines the DKG update before the next question, and Section 4.1.2 starts every dataset from an empty vector database. Since the main Table 1 experiments do not describe splitting the test set into an evolution split and an evaluation split, the natural reading is that WTS processes the test questions in order, with each gold answer feeding the DKG used for later questions. The headline 11.3% gain and the improvement-over-time plots are therefore not evidence about generalization to unseen questions; they are partly the model reading earlier answer keys. The one clean evaluation in Appendix Table 6 shows a much smaller gain. This is a benchmark-protocol flaw rather than a defect in the derivation of the method itself, but it makes the central SOTA claim unsupported as reported.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The ledger reveals that the central empirical claim depends on the gold-answer availability axiom and on several unspecified thresholds. No new physical or conceptual entities are introduced; the DKG is a data structure, not a postulated entity.

free parameters (5)
  • similarity threshold L = 0.55
    Set for all datasets; filters triples by cosine distance between question and triple embeddings.
  • redundancy threshold L' = not specified
    Used in Section 3.2.2 to suppress near-duplicate triples; value not reported.
  • pruning width K = not specified
    Number of triples retained at each depth; mentioned in Section 3.2.1 but value not reported.
  • max retrieval depth D = 2-4 per dataset
    Different per dataset (Table 5); appears tuned on data because Table 3 shows performance varies strongly with D.
  • number of extracted entities N = 5
    Entity extraction limits to 5 entities per question (Prompt 1).
assumptions (4)
  • domain assumption Gold answers are available during execution and can be used to generate triples
    Section 3.2.2 and 3.3 assume a mentor provides alpha*_q; in the main experiments this is the test-set answer key.
  • domain assumption Question ordering is fixed and arbitrary; early questions do not affect later ones beyond DKG content
    Ordering determines which gold-derived triples are available; no shuffling or variance analysis is reported.
  • domain assumption LLM-generated triples are correct and useful
    Triples are added after only a similarity-based redundancy check, with no verification of factual correctness (Section 3.2.2).
  • standard math Cosine similarity between embeddings reflects knowledge relevance
    Retrieval and redundancy checks use embedding cosine distance (Sections 3.2.1, 3.2.2).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Way to Specialist: Closing Loop Between Specialized LLM and Evolving Domain Knowledge Graph." pith.science (2026). https://pith.science/paper/4V7E3LKM

@misc{pith2026241119064,
  author       = {Pith},
  title        = {Pith review of: Way to Specialist: Closing Loop Between Specialized LLM and Evolving Domain Knowledge Graph},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4V7E3LKM}},
  note         = {Machine review of arXiv:2411.19064}
}
abstract

Large language models (LLMs) have demonstrated exceptional performance across a wide variety of domains. Nonetheless, generalist LLMs continue to fall short in reasoning tasks necessitating specialized knowledge. Prior investigations into specialized LLMs focused on domain-specific training, which entails substantial efforts in domain data acquisition and model parameter fine-tuning. To address these challenges, this paper proposes the Way-to-Specialist (WTS) framework, which synergizes retrieval-augmented generation with knowledge graphs (KGs) to enhance the specialized capability of LLMs in the absence of specialized training. In distinction to existing paradigms that merely utilize external knowledge from general KGs or static domain KGs to prompt LLM for enhanced domain-specific reasoning, WTS proposes an innovative "LLM$\circlearrowright$KG" paradigm, which achieves bidirectional enhancement between specialized LLM and domain knowledge graph (DKG). The proposed paradigm encompasses two closely coupled components: the DKG-Augmented LLM and the LLM-Assisted DKG Evolution. The former retrieves question-relevant domain knowledge from DKG and uses it to prompt LLM to enhance the reasoning capability for domain-specific tasks; the latter leverages LLM to generate new domain knowledge from processed tasks and use it to evolve DKG. WTS closes the loop between DKG-Augmented LLM and LLM-Assisted DKG Evolution, enabling continuous improvement in the domain specialization as it progressively answers and learns from domain-specific questions. We validate the performance of WTS on 6 datasets spanning 5 domains. The experimental results show that WTS surpasses the previous SOTA in 4 specialized domains and achieves a maximum performance improvement of 11.3%.

Figures

Figures reproduced from arXiv: 2411.19064 by the authors.

Figure 1
Figure 1. Comparison of “LLM⟳KG” against SOTA paradigms in KG-augmented LLM. have spurred research into specialized LLMs in medical field [18, 30, 49, 67], politics [35], scholar [53] and law [14]. These works utilize existing RAG frameworks, e.g. mind map [67], reason chain [18], and aligned embeddings [35], to generate input prompts. The works in the branch of LLM for KG leverages LLMs to support knowledge engineering tasks… view at source ↗
Figure 2
Figure 2. Overview of WTS. The upper part is DKG-Augmented LLM which extracts entities from the received question and performs iterative Retrieval-Prune-Reason processes. The bottom part is LLM-Assisted DKG Evolution which generates new knowledge triples from the processed question to evolve the DKG. to retrieve pertinent documents or data from an external source and use this information to augment the LLM’s predictions. A re… view at source ↗
Figure 3
Figure 3. Illustration of WTS formation pipeline. domain. Furthermore, this scenario may also occur when an expert utilizes WTS as an assistant and constantly provides gold answers to questions. The period of Mastership typically follows the Ap￾prenticeship stage, once a DKG has been sufficiently established for prompting domain-specific tasks. During Mastership, WTS oper￾ates autonomously without the guidance of gold answers… view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Retrieval time and execution time of WTS with GPT-3.5. ChatDoctor5k PubMedQA MedMCQA(Single) MedMCQA(Multi) SimpleQA sciq ScienceQA-NAT ScienceQA-SOC ScienceQA-LAN Dataset 0 500 1000 1500 2000 2500 3000 3500 KG Size WTS(GPT-3.5 w/CM) WTS(GPT-3.5 w/EM-ESR) WTS(GPT-3.5 w…
Figure 6
Figure 6. Figure 6: Percentage of questions across different retrieval [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Accuracy improvements and KG size in medical [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]
Figure 8
Figure 8. Figure 8: Proportions of WTS’s evidence of answers. [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

88 extracted references · 33 canonical work pages

  1. [1]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023)

    arXiv 2023

  2. [2]

    Dhruv Agarwal, Rajarshi Das, Sopan Khosla, and Rashmi Gangadharaiah. 2023. Bring your own kg: Self-supervised program synthesis for zero-shot kgqa. arXiv preprint arXiv:2311.07850 (2023)

    arXiv 2023

  3. [3]

    Dhananjay Ashok and Zachary C Lipton. 2023. Promptner: Prompting for named entity recognition. arXiv preprint arXiv:2305.15444 (2023)

    arXiv 2023

  4. [4]

    Jinheon Baek, Alham Fikri Aji, and Amir Saffari. 2023. Knowledge-Augmented Language Model Prompting for Zero-Shot Knowledge Graph Question Answer- ing. In The 61st Annual Meeting Of The Association For Computational Linguistics

    2023

  5. [5]

    Teodoro Baldazzi, Luigi Bellomarini, Stefano Ceri, Andrea Colombo, Andrea Gentili, and Emanuel Sallinger. 2023. Fine-tuning large enterprise language models via ontological reasoning. In International Joint Conference on Rules and Reasoning. Springer, 86–94

    2023

  6. [6]

    Zhen Bi, Jing Chen, Yinuo Jiang, Feiyu Xiong, Wei Guo, Huajun Chen, and Ningyu Zhang. 2024. Codekgc: Code language model for generative knowledge graph construction. ACM Transactions on Asian and Low-Resource Language Information Processing 23, 3 (2024), 1–16

    2024

  7. [7]

    Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor

  8. [8]

    Antoine Bordes, Nicolas Usunier, Sumit Chopra, and Jason Weston. 2015. Large- scale simple question answering with memory networks. arXiv preprint arXiv:1506.02075 (2015)

    arXiv 2015

Show all 88 references
  1. [9]

    Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Ok- sana Yakhnenko. 2013. Translating embeddings for modeling multi-relational data. Advances in neural information processing systems 26 (2013)

    2013

  2. [10]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey W...

    2020

  3. [11]

    Yihan Cao, Yanbin Kang, and Lichao Sun. 2023. Instruction mining: High- quality instruction data selection for large language models. arXiv preprint arXiv:2307.06290 (2023)

    2023 arXiv

  4. [12]

    Salvatore Carta, Alessandro Giuliani, Leonardo Piano, Alessandro Sebastian Podda, Livio Pompianu, and Sandro Gabriele Tiddia. 2023. Iterative zero-shot llm prompting for knowledge graph construction. arXiv preprint arXiv:2307.01128 (2023)

    2023 arXiv

  5. [13]

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Se- bastian Gehrmann, et al. 2023. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research 24, 240 (2023), 1–113

    2023

  6. [14]

    Jiaxi Cui, Zongjian Li, Yang Yan, Bohua Chen, and Li Yuan. 2023. Chatlaw: Open- source legal large language model with integrated external knowledge bases. arXiv preprint arXiv:2306.16092 (2023)

    2023 arXiv

  7. [15]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding.arXiv preprint arXiv:1810.04805 (2018)

    2018 arXiv

  8. [16]

    Steven Haussmann, Oshani Seneviratne, Yu Chen, Yarden Ne’eman, James Codella, Ching-Hua Chen, Deborah L McGuinness, and Mohammed J Zaki. 2019. FoodKG: a semantics-driven knowledge graph for food recommendation. In The Semantic Web–ISWC 2019: 18th International Semantic Web Con...

    2019

  9. [17]

    Yuelin Hu, Futai Zou, Jiajia Han, Xin Sun, and Yilei Wang. 2023. Llm-Tikg: Threat Intelligence Knowledge Graph Construction Utilizing Large Language Model. A vailable at SSRN 4671345 (2023)

    2023

  10. [18]

    Xinke Jiang, Ruizhe Zhang, Yongxin Xu, Rihong Qiu, Yue Fang, Zhiyuan Wang, Jinyi Tang, Hongxin Ding, Xu Chu, Junfeng Zhao, et al. 2023. Think and Retrieval: A Hypothesis Knowledge Graph Enhanced Medical Large Language Models. arXiv preprint arXiv:2312.15883 (2023)

    2023 arXiv

  11. [19]

    Zhouyu Jiang, Ling Zhong, Mengshu Sun, Jun Xu, Rui Sun, Hui Cai, Shuhan Luo, and Zhiqiang Zhang. 2024. Efficient Knowledge Infusion via KG-LLM Alignment. arXiv preprint arXiv:2406.03746 (2024)

    2024 arXiv

  12. [20]

    Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu

  13. [21]

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing S...

    2020

  14. [22]

    Haotian Li, Lingzhi Wang, Yuliang Wei, Richard Yi Da Xu, and Bailing Wang

  15. [23]

    Yunxiang Li, Zihan Li, Kai Zhang, Ruilong Dan, Steve Jiang, and You Zhang. 2023. Chatdoctor: A medical chat model fine-tuned on a large language model meta-ai 9 Conference acronym ’XX, June 03–05, 2018, Woodstock, NY Yutong Zhang, Lixing Chen, Shenghong Li, Nan Cao, Yang Shi, ...

    2023

  16. [24]

    Yong Lin, Lu Tan, Hangyu Lin, Zeming Zheng, Renjie Pi, Jipeng Zhang, Shizhe Diao, Haoxiang Wang, Han Zhao, Yuan Yao, et al. 2023. Speciality vs generality: An empirical study on catastrophic forgetting in fine-tuning foundation models. arXiv preprint arXiv:2309.06256 (2023)

    2023 arXiv

  17. [25]

    Qidong Liu, Xian Wu, Xiangyu Zhao, Yuanshao Zhu, Derong Xu, Feng Tian, and Yefeng Zheng. 2024. When MOE Meets LLMs: Parameter Efficient Fine-tuning for Multi-task Medical Applications. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in...

    2024

  18. [26]

    Weijie Liu, Peng Zhou, Zhe Zhao, Zhiruo Wang, Qi Ju, Haotang Deng, and Ping Wang. 2020. K-bert: Enabling language representation with knowledge graph. In Proceedings of the AAAI Conference on Artificial Intelligence , Vol. 34. 2901–2908

    2020

  19. [27]

    Zhengliang Liu, Aoxiao Zhong, Yiwei Li, Longtao Yang, Chao Ju, Zihao Wu, Chong Ma, Peng Shu, Cheng Chen, Sekeun Kim, et al . 2023. Tailoring large language models to radiology: A preliminary approach to llm adaptation for a highly specialized domain. In International Workshop ...

    2023

  20. [28]

    Linhao Luo, Yuan-Fang Li, Gholamreza Haffari, and Shirui Pan. 2023. Reasoning on graphs: Faithful and interpretable large language model reasoning. arXiv preprint arXiv:2310.01061 (2023)

    2023 arXiv

  21. [29]

    Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. 2022. When not to trust language models: Investigat- ing effectiveness of parametric and non-parametric memories. arXiv preprint arXiv:2212.10511 (2022)

    2022 arXiv

  22. [30]

    Nicholas Matsumoto, Jay Moran, Hyunjun Choi, Miguel E Hernandez, Mythreye Venkatesan, Paul Wang, and Jason H Moore. 2024. KRAGEN: a knowledge Graph- Enhanced RAG framework for biomedical problem solving using large language models. Bioinformatics (2024), btae353

    2024

  23. [31]

    Lars-Peter Meyer, Claus Stadler, Johannes Frey, Norman Radtke, Kurt Junghanns, Roy Meissner, Gordian Dziwis, Kirill Bulert, and Michael Martin. 2023. Llm- assisted knowledge graph engineering: Experiments with chatgpt. In Working conference on Artificial Intelligence Developme...

    2023

  24. [32]

    Grégoire Mialon, Roberto Dessì, Maria Lomeli, Christoforos Nalmpantis, Ram Pasunuru, Roberta Raileanu, Baptiste Rozière, Timo Schick, Jane Dwivedi-Yu, Asli Celikyilmaz, et al. 2023. Augmented language models: a survey. arXiv preprint arXiv:2302.07842 (2023)

    2023 arXiv

  25. [33]

    Nandana Mihindukulasooriya, Sanju Tiwari, Carlos F Enguix, and Kusum Lata

  26. [34]

    Sudha Morwal, Nusrat Jahan, and Deepti Chopra. 2012. Named entity recognition using hidden Markov model (HMM). International Journal on Natural Language Computing (IJNLC) Vol 1 (2012)

    2012

  27. [35]

    Xinyi Mou, Zejun Li, Hanjia Lyu, Jiebo Luo, and Zhongyu Wei. 2024. Unifying local and global knowledge: Empowering large language models as political experts with knowledge graphs. In Proceedings of the ACM on Web Conference

    2024

  28. [36]

    OpenAI. [n. d.]. Chat Completions Guide. https://platform.openai.com/docs/ guides/chat-completions. Accessed: 2024-08-06

    2024

  29. [37]

    In International Semantic Web Conference

    Text2kgbench: A benchmark for ontology-driven knowledge graph gener- ation from text. In International Semantic Web Conference. Springer, 247–265

  30. [38]

    Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. 2022. MedMCQA: A Large-scale Multi-Subject Multi-Choice Dataset for Medical do- main Question Answering. In Proceedings of the Conference on Health, Inference, and Learning (Proceedings of Machine Learning Researc...

    2022

  31. [39]

    Jeff Z Pan, Simon Razniewski, Jan-Christoph Kalo, Sneha Singhania, Jiaoyan Chen, Stefan Dietze, Hajira Jabeen, Janna Omeliyanenko, Wen Zhang, Matteo Lis- sandrini, et al. 2023. Large language models and knowledge graphs: Opportunities and challenges. arXiv preprint arXiv:2308....

    2023 arXiv

  32. [40]

    Nita Patil, Ajay Patil, and BV Pawar. 2020. Named entity recognition using conditional random fields. Procedia Computer Science 167 (2020), 1181–1188

    2020

  33. [41]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems 35...

    2022

  34. [42]

    André Gomes Regino, Rodrigo Oliveira Caus, Victor Hochgreb, and Julio Cesar dos Reis. 2022. From Natural Language Texts to RDF Triples: A Novel Approach to Generating e-Commerce Knowledge Graphs. In International Joint Conference on Knowledge Discovery, Knowledge Engineering, ...

    2022

  35. [43]

    Ali Sadeghian, Mohammadreza Armandpour, Patrick Ding, and Daisy Zhe Wang

  36. [44]

    Tanik Saikh, Tirthankar Ghosal, Amish Mittal, Asif Ekbal, and Pushpak Bhat- tacharyya. 2022. Scienceqa: A novel resource for question answering on scholarly articles. International Journal on Digital Libraries 23, 3 (2022), 289–301

    2022

  37. [45]

    Baolin Peng, Michel Galley, Pengcheng He, Hao Cheng, Yujia Xie, Yu Hu, Qiuyuan Huang, Lars Liden, Zhou Yu, Weizhu Chen, et al. 2023. Check your facts and try again: Improving large language models with external knowledge and automated feedback. arXiv preprint arXiv:2302.12813 (2023)

    2023 arXiv

  38. [46]

    Apoorv Saxena, Aditay Tripathi, and Partha Talukdar. 2020. Improving multi-hop question answering over knowledge graphs using knowledge base embeddings. In Proceedings of the 58th annual meeting of the association for computational linguistics. 4498–4507

    2020

  39. [47]

    Viktor Schlegel and André Freitas. 2019. DBee: A database for creating and managing knowledge graphs and embeddings. In Proceedings of the Thirteenth Workshop on Graph-Based Methods for Natural Language Processing (TextGraphs- 13). 177–185

    2019

  40. [48]

    Ad- vances in Neural Information Processing Systems 32 (2019)

    Drum: End-to-end differentiable rule mining on knowledge graphs. Ad- vances in Neural Information Processing Systems 32 (2019)

    2019

  41. [49]

    Karthik Soman, Peter W Rose, John H Morris, Rabia E Akbas, Brett Smith, Braian Peetoom, Catalina Villouta-Reyes, Gabriel Cerono, Yongmei Shi, Angela Rizk- Jackson, et al. 2023. Biomedical knowledge graph-enhanced prompt generation for large language models. arXiv preprint arXi...

    2023 arXiv

  42. [50]

    Prateek Sancheti, Kamalakar Karlapalem, and Kavita Vemuri. 2024. LLM Driven Web Profile Extraction for Identical Names. InCompanion Proceedings of the ACM on Web Conference 2024. 1616–1625

    2024

  43. [51]

    Jiashuo Sun, Chengjin Xu, Lumingyuan Tang, Saizhuo Wang, Chen Lin, Yeyun Gong, Heung-Yeung Shum, and Jian Guo. 2023. Think-on-graph: Deep and responsible reasoning of large language model with knowledge graph. arXiv preprint arXiv:2307.07697 (2023)

    2023 arXiv

  44. [52]

    Tianxiang Sun, Yunfan Shao, Xipeng Qiu, Qipeng Guo, Yaru Hu, Xuanjing Huang, and Zheng Zhang. 2020. Colake: Contextualized language and knowledge em- bedding. arXiv preprint arXiv:2010.00309 (2020)

    2020 arXiv

  45. [53]

    Kurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela, and Jason Weston. 2021. Retrieval augmentation reduces hallucination in conversation. arXiv preprint arXiv:2104.07567 (2021)

    2021 arXiv

  46. [54]

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 (2023)

    2023 arXiv

  47. [55]

    Jingyun Sun, Chengxiao Dai, Zhongze Luo, Yangbo Chang, and Yang Li

  48. [56]

    Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kul- shreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. 2022. Lamda: Language models for dialog applications. arXiv preprint arXiv:2201.08239 (2022)

    2022 arXiv

  49. [57]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023)

    2023 arXiv

  50. [58]

    Théo Trouillon, Johannes Welbl, Sebastian Riedel, Éric Gaussier, and Guillaume Bouchard. 2016. Complex embeddings for simple link prediction. In International conference on machine learning . PMLR, 2071–2080

    2016

  51. [59]

    Tilahun Abedissa Taffa and Ricardo Usbeck. 2023. Leveraging llms in scholarly knowledge graph question answering. arXiv preprint arXiv:2311.09841 (2023)

    2023 arXiv

  52. [60]

    Bo Wang, Tao Shen, Guodong Long, Tianyi Zhou, Ying Wang, and Yi Chang

  53. [61]

    Arun James Thirunavukarasu, Darren Shu Jeng Ting, Kabilan Elangovan, Laura Gutierrez, Ting Fang Tan, and Daniel Shu Wei Ting. 2023. Large language models in medicine. Nature medicine 29, 8 (2023), 1930–1940

    2023

  54. [62]

    Shuhe Wang, Xiaofei Sun, Xiaoya Li, Rongbin Ouyang, Fei Wu, Tianwei Zhang, Jiwei Li, and Guoyin Wang. 2023. Gpt-ner: Named entity recognition via large language models. arXiv preprint arXiv:2304.10428 (2023)

    2023 arXiv

  55. [63]

    Xiaoyan Wang, Pavan Kapanipathi, Ryan Musa, Mo Yu, Kartik Talamadupula, Ibrahim Abdelaziz, Maria Chang, Achille Fokoue, Bassem Makni, Nicholas Mattei, et al. 2019. Improving natural language inference using external knowledge in the science questions domain. In Proceedings of ...

    2019

  56. [64]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems 35 (2022), 24824–24837

    2022

  57. [65]

    Denny Vrandečić and Markus Krötzsch. 2014. Wikidata: a free collaborative knowledgebase. Commun. ACM 57, 10 (2014), 78–85

    2014

  58. [66]

    Liu, and Matt Gardner

    Johannes Welbl, Nelson F. Liu, and Matt Gardner. 2017. Crowdsourcing Mul- tiple Choice Science Questions. ArXiv abs/1707.06209 (2017). https://api. 10 Way to Specialist: Closing Loop Between Specialized LLM and Evolving Domain Knowledge Graph Conference acronym ’XX, June 03–05...

    2017 arXiv

  59. [67]

    Yilin Wen, Zifeng Wang, and Jimeng Sun. 2023. Mindmap: Knowledge graph prompting sparks graph of thoughts in large language models. arXiv preprint arXiv:2308.09729 (2023)

    2023 arXiv

  60. [68]

    Liang Wang, Wei Zhao, Zhuoyu Wei, and Jingming Liu. 2022. SimKGC: Simple Contrastive Knowledge Graph Completion with Pre-trained Language Models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) . 4281–4294

    2022

  61. [69]

    Fei Xia, Bin Li, Yixuan Weng, Shizhu He, Kang Liu, Bin Sun, Shutao Li, and Jun Zhao. 2022. MedConQA: medical conversational question answering system based on knowledge graphs. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: System De...

    2022

  62. [70]

    Tianbao Xie, Chen Henry Wu, Peng Shi, Ruiqi Zhong, Torsten Scholak, Michihiro Yasunaga, Chien-Sheng Wu, Ming Zhong, Pengcheng Yin, Sida I Wang, et al

  63. [71]

    Yao Xu, Shizhu He, Jiabei Chen, Zihao Wang, Yangqiu Song, Hanghang Tong, Kang Liu, and Jun Zhao. 2024. Generate-on-Graph: Treat LLM as both Agent and KG in Incomplete Knowledge Graph Question Answering. arXiv preprint arXiv:2404.14741 (2024)

    2024 arXiv

  64. [72]

    Yanbin Wei, Qiushi Huang, James T Kwok, and Yu Zhang. 2024. KICGPT: Large Language Model with Knowledge in Context for Knowledge Graph Completion. arXiv preprint arXiv:2402.02389 (2024)

    2024 arXiv

  65. [73]

    Qimin Yang, Rongsheng Wang, Jiexin Chen, Runqi Su, and Tao Tan. 2024. Fine- Tuning Medical Language Models for Enhanced Long-Contextual Understanding and Domain Expertise. arXiv preprint arXiv:2407.11536 (2024)

    2024 arXiv

  66. [74]

    Liang Yao, Chengsheng Mao, and Yuan Luo. 2019. KG-BERT: BERT for knowledge graph completion. arXiv preprint arXiv:1909.03193 (2019)

    2019 arXiv

  67. [75]

    Yike Wu, Nan Hu, Guilin Qi, Sheng Bi, Jie Ren, Anhuan Xie, and Wei Song

  68. [76]

    arXiv preprint arXiv:2309.11206 (2023)

    Retrieve-rewrite-answer: A KG-to-text enhanced LLMS framework for knowledge graph question answering. arXiv preprint arXiv:2309.11206 (2023)

    2023 arXiv

  69. [77]

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675 (2019)

    2019 arXiv

  70. [78]

    GuoDong Zhou and Jian Su. 2002. Named entity recognition using an HMM- based chunk tagger. In Proceedings of the 40th annual meeting of the association for computational linguistics. 473–480. A ALGORITHM FOR WTS We present a detailed overview of the algorithmic progress of WTS...

    2002

  71. [81]

    Fan Yang, Zhilin Yang, and William W Cohen. 2017. Differentiable learning of logical rules for knowledge base reasoning. Advances in neural information processing systems 30 (2017)

    2017

  72. [84]

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. 2024. Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems 36 (2024)

    2024

  73. [85]

    Miao Zhang, Rufeng Dai, Ming Dong, and Tingting He. 2022. Drlk: dynamic hierarchical reasoning with language model and knowledge graph for question answering. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 5123–5133

    2022

  74. [88]

    Question Entity Extraction

    To avoid the influence of context provided by the dataset on the 11 Conference acronym ’XX, June 03–05, 2018, Woodstock, NY Yutong Zhang, Lixing Chen, Shenghong Li, Nan Cao, Yang Shi, Jiaxin Ding, Zhe Qu, Pan Zhou, and Yang Bai evaluation of methods, no extra context informati...

    2018

  75. [2008]

    In Proceedings of the 2008 ACM SIGMOD international conference on Management of data

    Freebase: a collaboratively created graph database for structuring human knowledge. In Proceedings of the 2008 ACM SIGMOD international conference on Management of data. 1247–1250

    2008

  76. [2019]

    In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)

    PubMedQA: A Dataset for Biomedical Research Question Answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2567–2577

    2019

  77. [2021]

    In Proceedings of the Web Conference 2021

    Structure-augmented text representation learning for efficient knowledge graph completion. In Proceedings of the Web Conference 2021 . 1737–1748

    2021

  78. [2022]

    arXiv preprint arXiv:2201.05966 (2022)

    Unifiedskg: Unifying and multi-tasking structured knowledge grounding with text-to-text language models. arXiv preprint arXiv:2201.05966 (2022)

    2022 arXiv

  79. [2023]

    arXiv preprint arXiv:2309.14770 (2023)

    KERMIT: Knowledge Graph Completion of Enhanced Relation Modeling with Inverse Transformation. arXiv preprint arXiv:2309.14770 (2023)

    2023 arXiv

  80. [2024]

    arXiv preprint arXiv:2407.16252 (2024)

    LawLuo: A Chinese Law Firm Co-run by LLM Agents. arXiv preprint arXiv:2407.16252 (2024)

    2024 arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.