Pith. sign in

REVIEW 4 major objections 5 minor 33 references

SPARQL Query Generation with LLMs: Measuring the Impact of Training Data Memorization and Knowledge Injection

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims LLMs lean on memorized training data when generating SPARQL queries, and offers a masked-injection protocol to measure the memorization share, warning that familiar-benchmark results will not transfer to private or new…

desk verdict A genuinely useful masked-injection protocol, but the memorization claim rests on a confounded dataset comparison and internally inconsistent tables. read the letter →

arxiv 2507.13859 v1 pith:XGH5QKV4 submitted 2025-07-18 cs.IR cs.AIcs.CL

Aleksandr Gashkov , Aleksandr Perevalov , Maria Eltsova , Andreas Both This is my paper
classification cs.IRcs.AIcs.CL
keywords SPARQLquerygenerationlargelanguagemodelsknowledgegraphquestionansweringLLMmemorizationinjectionQALD-9-plusMCWQWikidata
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

LLMs are increasingly used to translate natural-language questions into SPARQL queries over knowledge graphs, but because they are trained on web-scale data, researchers cannot tell whether a good score reflects reasoning or simple recall of a benchmark that was already in the training set. This paper tries to isolate the memorization channel by evaluating query generation under three conditions: zero-shot prompting, prompting with complete entity/property mappings (knowledge injection), and prompting with the same mappings but anonymized URIs and no mention of Wikidata (masked knowledge injection). The authors find that performance on the widely used QALD-9-plus benchmark far exceeds performance on the rarely used MCWQ benchmark, and that under masked injection many models still emit correct Wikidata URIs. They conclude that LLMs rely heavily on memorized training data when generating SPARQL, and that results on familiar benchmarks are unlikely to be reproducible on private or new datasets. The proposed protocol is portable to any knowledge graph and any LLM, offering a way to estimate how much of a reported quality figure is memorization.

What carries the argument

The load-bearing instrument is the masked-knowledge-injection prompt: the model is given the natural-language question plus exact label-to-URI mappings, but the URIs are anonymized as kg:NNNN and the prompt refers only to 'a Knowledge Graph', never to Wikidata. The reasoning is that a model that truly composes the query from the provided mappings will succeed equally under masked and unmasked injection, whereas a model that falls back on memorized examples will produce either the original Wikidata URIs (Category 4 leakage) or otherwise degrade. The paired datasets, QALD-9-plus (popular) and MCWQ (rare), both over Wikidata, provide the contrast in training-data exposure that makes the memorization signal measurable.

What would settle it

Run the masked-injection protocol on a newly created, never-publicly-released Wikidata benchmark whose question complexity is matched to QALD-9-plus; if accuracy stays at QALD-9-plus levels while Wikidata-URI leakage disappears, the paper's memorization explanation would be defeated.

Watch

Extended reading notes

Core claim

The paper's central claim is that LLMs' apparent competence at SPARQL query generation is substantially an artifact of training-data memorization rather than compositional reasoning. The evidence is twofold: first, all tested models perform much worse on MCWQ, an infrequently used benchmark over Wikidata, than on the popular QALD-9-plus dataset even when the prompt supplies every entity and property mapping; second, in the masked-knowledge-injection condition, in which Wikidata URIs are replaced by random kg: identifiers and the prompt never names Wikidata, a substantial share of generated queries still contain the correct original Wikidata URIs. The authors interpret this as Error Category 4: the model is retrieving a memorized gold query rather than composing from the prompt. They further show that reformulating MCWQ's linguistically awkward questions does not close the gap, which they take as evidence that dataset memorization, not question phrasing, drives the performance difference.

Load-bearing premise

The conclusion depends on the premise that MCWQ was not memorized by the models and that the two datasets are comparable in complexity, so that the higher QALD-9-plus scores are attributable to memorization rather than to MCWQ being intrinsically harder.

Editorial extensions

If this is right

  • Knowledge injection with complete entity and property mappings substantially improves SPARQL generation for nearly all tested models, while zero-shot generation is mostly poor.
  • Performance on the popular QALD-9-plus benchmark is consistently higher than on the rarely used MCWQ benchmark, even with identical injection quality.
  • Under masked knowledge injection, models frequently emit correct Wikidata URIs despite being instructed to use an unknown graph, indicating recall of specific training examples.
  • Consequently, state-of-the-art reported results on familiar benchmarks are unlikely to transfer to private or novel knowledge graphs: the memorization effect inflates quality on known data.
  • The evaluation protocol itself is portable and can be applied to any knowledge graph and any LLM, providing a practical way to audit how much of a result is memorization.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension is to treat the masked-injection leakage rate as a membership-inference probe: a model that emits the exact Wikidata URI for an anonymized pair is strong evidence that the underlying fact or query template was in its training data.
  • The paper's results suggest that retrieval-augmented prompting with entity mappings can lift small models closer to large ones, but the leakage phenomenon warns that such gains may still be anchored in memorized queries rather than genuine compositional skill.
  • Because reformulating MCWQ did not close the gap, the confound between memorization and compositional difficulty is not fully resolved; a follow-up with synthetic, complexity-matched questions with controlled novelty would separate the two effects.
  • Applying the same three-condition protocol to commercial models and to non-English questions would test whether the memorization pattern generalizes beyond the open-source, English-only setting studied here.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes an evaluation protocol for LLM-based SPARQL query generation that compares three prompting conditions: zero-shot, knowledge injection, and masked knowledge injection. The method is applied to 11 open-weight LLMs on two Wikidata-backed benchmarks, QALD-9-plus and MCWQ, with quality measured by valid-query rate (Pval) and F1. The authors report that knowledge injection substantially improves performance, that models produce Wikidata URIs even under masked prompting, and that performance on the popular QALD-9-plus far exceeds that on the rarely used MCWQ. From these results they conclude that LLMs rely heavily on memorized training data rather than reasoning, and that LLM-based SPARQL generation is unlikely to be fully reproducible on private or new datasets.

Significance. If the central claim were established, the proposed protocol would be a practical tool for estimating benchmark-contamination effects in KGQA, and the finding would be important for the community. The paper has concrete strengths: a reproducible setup with code and data released, a broad model sweep from 7B to 123B, and a clean three-condition prompting design that separates the effect of providing identifiers from providing anonymized ones. The error analysis is a useful descriptive addition. However, the memorization conclusion currently rests on a confounded dataset comparison, an inconsistent table, and an unverified training-data assumption, so the main inference is not yet supported.

major comments (4)
  1. [Sec. 3.2 / Sec. 4.1 / Sec. 6] The central attribution of the QALD-9-plus vs. MCWQ performance gap to memorization is confounded by query complexity. The authors state in Sec. 3.2 that MCWQ questions 'express multiple restrictions ... sometimes much more restrictions than QALD-9-plus has'; with knowledge injection, the model must still assemble those restrictions correctly, so structural difficulty alone can produce lower F1 on MCWQ. The optimized-MCWQ experiment in Sec. 4.1 only reformulates 140 questions to improve language quality and does not alter the number of triple patterns or joins, so it does not rule out the complexity explanation. Because Sec. 6 uses this gap as the main evidence for the conclusion that results are 'unlikely to be fully reproducible' on private or new datasets, the authors should either match the datasets on complexity (e.g., by number of triple patterns, joins, or query shapes) or substantially weaken this claim.
  2. [Tables 2 and 3] The Pval column appears internally inconsistent: for a given model the same Pval is reported across the three prompting conditions even though the number of valid queries differs. For example, Qwen 2.5 7B on MCWQ has 55, 81, and 90 valid queries but Pval=0.35 in all three columns, and Qwen 2.5 72B on QALD-9-plus has 247, 400, and 425 valid queries but Pval=0.52 in all three columns. Since Sec. 3.2 defines Pval as the relative frequency of valid queries, these entries cannot all be correct; the tables need to be corrected before the reported QALD-vs-MCWQ comparisons can be assessed.
  3. [Sec. 3.2] The conclusion that MCWQ is less memorized than QALD-9-plus rests on the unverified assumption that MCWQ was absent from, or at least much less present in, LLM pretraining data. The text says 'we suppose that there is a higher tendency of LLMs to memorize popular datasets,' but no contamination check, membership test, or temporal analysis is supplied. Without evidence about training-data presence, the performance gap could reflect frequency effects, dataset construction differences, or other properties. Please report a concrete check (e.g., probing with dataset-specific prefixes, comparing release dates with training-data cutoffs, or using a contamination-detection method) or temper the claim to a hypothesis.
  4. [Sec. 4.2 / Sec. 5] The masked-injection evidence for memorization is interpreted too strongly. Sec. 4.2 and Sec. 5 treat the occurrence of any Wikidata URI under masked prompting as evidence that the model recalled the specific gold query, but a model with a general bias toward the Wikidata prefix would also emit wd: URIs. The analysis should distinguish exact correct URIs from the gold query, plausible but incorrect Wikidata URIs, and invented URIs; Table 4 aggregates Error Category 4 without making this distinction, so it does not quantify memorization of specific training examples.
minor comments (5)
  1. [Sec. 4.1] The text contains a typo: 'MCQW' should be 'MCWQ' in the sentence 'only the top result in the same experiment on MCQW only aF1 = 0.15was achieved by Qwen 2.5 72B.'
  2. [Table 2] The model name 'qwen2.5 14B' is lowercase in the QALD-9-plus block, inconsistent with the capitalization used elsewhere in the table.
  3. [Table 4] The error frequencies are reported without the underlying counts or denominators; adding the total number of erroneous queries would make the rates interpretable.
  4. [Sec. 3.2 / Table 2] The total number of QALD-9-plus records differs between zero-shot (471) and the two injection conditions (460); the paper does not explain why the dataset size changes, and this should be clarified.
  5. [Tables 2 and 3] No confidence intervals or repeated-run variability is reported; given that several F1 differences in Table 3 are small (e.g., 0.09 vs. 0.05 for Qwen 2.5 7B), a statement about run-to-run variance would help the reader judge the robustness of the comparisons.

Circularity Check

0 steps flagged · score 1.0 of 10

No construction-level circularity; the memorization claim rests on an unverified comparability assumption, which is a validity risk, not a circular derivation.

full rationale

This is an empirical measurement study, not a derivation. The three prompting conditions are defined independently of the outcome: zero-shot, knowledge injection, and masked knowledge injection are distinct inputs, and the F1/Pval scores are computed from LLM outputs against gold-standard SPARQL queries from external benchmarks (QALD-9-plus and MCWQ). No parameter is fitted to the target result and then renamed a prediction. The QALD-9-plus dataset [18] is a self-citation (Perevalov et al.), but it is a public, externally maintained benchmark and is not used as evidence for the memorization claim beyond providing test questions. The central inference that the QALD-vs-MCWQ gap and the occurrence of Wikidata URIs under masked prompts indicate memorization rests on the unverified assumptions that MCWQ is absent from training data and that the datasets are comparable in structural difficulty; those are threats to the validity of the conclusion, not cases where the conclusion is equivalent to the input by construction. The paper's own statement that MCWQ questions 'sometimes much more restrictions than QALD-9-plus has' (Sec. 3.2) exposes a confound, but a confounded empirical inference is not circularity under the criteria used here. Therefore no circular step can be exhibited, and the score is 1 rather than 0 only because the dataset citation involves author overlap.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No fitted parameters or new entities. The empirical protocol introduces no mathematical claims; the main dependencies are domain assumptions about dataset correctness, sample representativeness, and the unverified training-data absence of MCWQ.

assumptions (4)
  • domain assumption Gold standard SPARQL queries in QALD-9-plus and MCWQ are correct and executable over Wikidata, and a non-empty answer exists for every test question.
    The metrics precision, recall, and F1 are asserted equal only if each question has a non-empty gold answer; this is assumed throughout Section 3.2.
  • domain assumption The knowledge injection condition provides all information needed to generate a correct query, and entity linking from gold queries is perfect.
    Used in Section 3.1 to define the 'perfect knowledge injection' prompt.
  • domain assumption The tested subsets (155 MCWQ questions, 471 or 460 QALD-9-plus questions) are representative samples of the benchmarks.
    No sampling procedure is described; the total MCWQ has 124,187 pairs, so 155 is a small uncharacterized sample.
  • ad hoc to paper MCWQ was absent from LLM pretraining data, or at least much less memorized than QALD-9-plus.
    The central RQ2 comparison depends on this premise; Section 3.2 only 'supposes' a higher tendency to memorize popular datasets without verifying.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SPARQL Query Generation with LLMs: Measuring the Impact of Training Data Memorization and Knowledge Injection." pith.science (2026). https://pith.science/paper/XGH5QKV4

@misc{pith2026250713859,
  author       = {Pith},
  title        = {Pith review of: SPARQL Query Generation with LLMs: Measuring the Impact of Training Data Memorization and Knowledge Injection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XGH5QKV4}},
  note         = {Machine review of arXiv:2507.13859}
}
read the original abstract

Nowadays, the importance of software with natural-language user interfaces cannot be underestimated. In particular, in Question Answering (QA) systems, generating a SPARQL query for a given natural-language question (often named Query Building) from the information retrieved from the same question is the central task of QA systems working over Knowledge Graphs (KGQA). Due to the rise of Large Language Models (LLMs), they are considered a well-suited method to increase the quality of the question-answering functionality, as there is still a lot of room for improvement, aiming for enhanced quality and trustworthiness. However, LLMs are trained on web data, where researchers have no control over whether the benchmark or the knowledge graph was already included in the training data. In this paper, we introduce a novel method that evaluates the quality of LLMs by generating a SPARQL query from a natural-language question under various conditions: (1) zero-shot SPARQL generation, (2) with knowledge injection, and (3) with "anonymized" knowledge injection. This enables us, for the first time, to estimate the influence of the training data on the QA quality improved by LLMs. Ultimately, this will help to identify how portable a method is or whether good results might mostly be achieved because a benchmark was already included in the training data (cf. LLM memorization). The developed method is portable, robust, and supports any knowledge graph; therefore, it could be easily applied to any KGQA or LLM, s.t., generating consistent insights into the actual LLM capabilities is possible.

Figures

Figures reproduced from arXiv: 2507.13859 by the authors.

Figure 1
Figure 1. Overview of the proposed evaluation approach and prompting strategies [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Examples of prompts to generate the SPARQL query for the question [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Examples from the identified error categories in the generated SPARQL [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

33 extracted references · 28 canonical work pages

  1. [1]

    Available at SSRN 5009450

    Bhandiwad, D., Gattogi, P., Kangen, A., Basaldella, M., Ferré, S., Vahdati, S., Lehmann, J.: Bridging language models and knowledge graphs with controlled natural languages. Available at SSRN 5009450

    work page

  2. [2]

    Leveraging small language models for Text2SPARQL tasks to improve the resilience of AI assistance

    Brei,F.,Frey,J.,Meyer,L.P.:LeveragingsmalllanguagemodelsforText2SPARQL tasks to improve the resilience of AI assistance. arXiv:2405.17076 (2024)

    work page Pith review arXiv 2024

  3. [3]

    Transactions of the ACL10(2022)

    Cui, R., Aralikatte, R., Lent, H., Hershcovich, D.: Compositional generalization in multilingual semantic parsing over Wikidata. Transactions of the ACL10(2022)

    work page 2022

  4. [4]

    IEEE Access (2024)

    Diallo, P.A.K.K., Reyd, S., Zouaq, A.: A comprehensive evaluation of neural SPARQL query generation from natural language questions. IEEE Access (2024)

    work page 2024

  5. [5]

    In: Interna- tional semantic web conference

    Dubey, M., Banerjee, D., Abdelkawi, A., Lehmann, J.: LC-QuAD 2.0: A large dataset for complex question answering over Wikidata and DBpedia. In: Interna- tional semantic web conference. pp. 69–78. Springer (2019)

    work page 2019

  6. [6]

    arXiv preprint arXiv:2410.06062 (2024)

    Emonet, V., Bolleman, J., Duvaud, S., de Farias, T.M., Sima, A.C.: LLM-based SPARQL query generation from natural language over federated knowledge graphs. arXiv preprint arXiv:2410.06062 (2024)

    arXiv 2024

  7. [7]

    Open Access Series in Informatics (OASIcs)113, 1–15 (2023)

    Faria, B., Perdigão, D., Oliveira, H.G.: Question answering over linked data with GPT-3. Open Access Series in Informatics (OASIcs)113, 1–15 (2023)

    work page 2023

  8. [8]

    arXiv preprint arXiv:2501.12948 (2025)

    Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang, P., Bi, X., et al.: DeepSeek-r1: Incentivizing reasoning capability in LLMs via reinforcement learning. arXiv preprint arXiv:2501.12948 (2025)

    arXiv 2025

Show all 33 references
  1. [9]

    In: The Semantic Web

    Kacupaj, E., Zafar, H., Lehmann, J., Maleshkova, M.: VQuAnDa: Verbalization question answering dataset. In: The Semantic Web. pp. 531–547. Springer Inter- national Publishing, Cham (2020)

    2020

  2. [10]

    In: SEMANTiCS (Posters & Demos) (2023)

    Kovriguina, L., Teucher, R., Radyush, D., Mouromtsev, D., Keshan, N., Neumaier, S., Gentile, A., Vahdati, S.: SPARQLGEN: One-shot prompt-based approach for SPARQL query generation. In: SEMANTiCS (Posters & Demos) (2023)

    2023

  3. [11]

    Transactions of the ACL12, 786–802 (2024)

    Lehmann, J., Bhandiwad, D., Gattogi, P., Vahdati, S.: Beyond boundaries: A human-like approach for question answering over structured and unstructured in- formation sources. Transactions of the ACL12, 786–802 (2024)

    2024

  4. [12]

    In: ECAI 2023, pp

    Lehmann, J., Gattogi, P., Bhandiwad, D., Ferré, S., Vahdati, S.: Language mod- els as controlled natural language semantic parsers for knowledge graph question answering. In: ECAI 2023, pp. 1348–1356. IOS Press (2023)

    2023

  5. [13]

    In: European Semantic Web Conference

    Lehmann, J., Meloni, A., Motta, E., Osborne, F., Recupero, D.R., Salatino, A.A., Vahdati, S.: Large language models for scientific question answering: An extensive analysis of the SciQA benchmark. In: European Semantic Web Conference. pp. 199–217. Springer (2024)

    2024

  6. [14]

    In: Findings of the Association for Computational Linguistics ACL 2024

    Liu,J.,Cao,S.,Shi,J.,Zhang,T.,Nie,L.,Hu,L.,Hou,L.,Li,J.:Howproficientare large language models in formal languages? An in-depth insight for knowledge base question answering. In: Findings of the Association for Computational Linguistics ACL 2024. pp. 792–815 (2024)

    2024

  7. [15]

    In: Findings of the Association for Computational Linguistics: EMNLP 2024

    Liu, S., Semnani, S., Triedman, H., Xu, J., Zhao, I.D., Lam, M.: SPINACH: SPARQL-based information navigation for challenging real-world questions. In: Findings of the Association for Computational Linguistics: EMNLP 2024. pp. 15977–16001. Association for Computational Linguis...

    2024

  8. [16]

    In: Proceedings of the Workshop on Generative AI and Knowledge Graphs (GenAIK)

    Mecharnia, T., d’Aquin, M.: Performance and limitations of fine-tuned LLMs in SPARQL query generation. In: Proceedings of the Workshop on Generative AI and Knowledge Graphs (GenAIK). pp. 69–77 (2025) Preprint of publication atInternational Conference on Web Engineering 2025 (I...

    2025

  9. [17]

    arXiv preprint arXiv:2409.05925 (2024)

    Meyer, L.P., Frey, J., Brei, F., Arndt, N.: Assessing SPARQL capabilities of large language models. arXiv preprint arXiv:2409.05925 (2024)

    2024 arXiv

  10. [18]

    In: 2022 IEEE 16th International Conference on Semantic Computing (ICSC)

    Perevalov, A., Diefenbach, D., Usbeck, R., Both, A.: QALD-9-plus: A multilingual dataset for question answering over DBpedia and Wikidata translated by native speakers. In: 2022 IEEE 16th International Conference on Semantic Computing (ICSC). pp. 229–234 (2022)

    2022

  11. [19]

    In: International Conference on Web Engineering

    Perevalov, A., Gashkov, A., Eltsova, M., Both, A.: Language models as SPARQL query filtering for improving the quality of multilingual question answering over knowledge graphs. In: International Conference on Web Engineering. pp. 3–18. Springer (2024)

    2024

  12. [20]

    In: International Semantic Web Conference

    Perevalov, A., Gashkov, A., Eltsova, M., Both, A.: Understanding SPARQL queries: Are we already there? Multilingual natural language generation based on SPARQL queries and large language models. In: International Semantic Web Conference. pp. 173–191. Springer (2024)

    2024

  13. [21]

    arXiv:2405.12819 (2024)

    Qin, L., Chen, Q., Feng, X., Wu, Y., Zhang, Y., Li, Y., Li, M., Che, W., Yu, P.S.: Large language models meet NLP: A survey. arXiv:2405.12819 (2024)

    2024 arXiv

  14. [22]

    SWAT4HCLS 2024: The 15th International Conference on Se- mantic Web Applications and Tools for Health Care and Life Sciences (2024)

    Rangel, J.C., de Farias, T.M., Sima, A.C., Kobayashi, N.: SPARQL generation: an analysis on fine-tuning OpenLLaMA for question answering over a life science knowledge graph. SWAT4HCLS 2024: The 15th International Conference on Se- mantic Web Applications and Tools for Health C...

    2024

  15. [23]

    In: Proceedings of the 31st International Conference on Computational Linguistics

    Shen, T., Wang, J., Zhang, X., Cambria, E.: Reasoning with trees: Faithful ques- tion answering over knowledge graph. In: Proceedings of the 31st International Conference on Computational Linguistics. pp. 3138–3157 (2025)

    2025

  16. [24]

    Soruco, J., Collarana, D., Both, A., Usbeck, R.: QALD-9-ES: A Spanish Dataset for Question Answering Systems, pp. 38–52. Studies on the Semantic Web (2023)

    2023

  17. [25]

    ACM Transactions on Computing Education (TOCE)18(3), 1–29 (2018)

    Taipalus, T., Siponen, M., Vartiainen, T.: Errors and complications in sql query formulation. ACM Transactions on Computing Education (TOCE)18(3), 1–29 (2018)

    2018

  18. [26]

    Semantic Web15(6), 2193–2207 (2024)

    Usbeck, R., Yan, X., Perevalov, A., Jiang, L., Schulz, J., Kraft, A., Möller, C., Huang, J., Reineke, J., Ngonga Ngomo, A.C., et al.: QALD-10–the 10th challenge on question answering over linked data: Shifting from DBpedia to Wikidata as a KG for KGQA. Semantic Web15(6), 2193–...

    2024

  19. [27]

    Advances in neural information processing systems35, 24824–24837 (2022)

    Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q.V., Zhou, D., et al.: Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems35, 24824–24837 (2022)

    2022

  20. [28]

    arXiv preprint arXiv:2412.15115 (2024)

    Yang,A.,Yang,B.,Zhang,B.,Hui,B.,Zheng,B.,Yu,B.,Li,C.,Liu,D.,Huang,F., Wei, H., et al.: Qwen2.5 technical report. arXiv preprint arXiv:2412.15115 (2024)

    2024 arXiv

  21. [29]

    Information12(7), 271 (2021)

    Yani, M., Krisnadhi, A.A.: Challenges, techniques, and trends of simple knowledge graph question answering: a survey. Information12(7), 271 (2021)

    2021

  22. [30]

    SEMANTiCS (2024)

    Zahera, H.M., Ali, M., Sherif, M.A., Moussallem, D., Ngomo, A.C.N.: Generating SPARQL from natural language using chain-of-thoughts prompting. SEMANTiCS (2024)

    2024

  23. [31]

    In: 31st Interna- tional Conference on Computational Linguistics

    Zhang, Z., Wen, L., Zhao, W.: Rule-KBQA: Rule-guided reasoning for complex knowledge base question answering with large language models. In: 31st Interna- tional Conference on Computational Linguistics. pp. 8399–8417 (2025)

    2025

  24. [32]

    Advances in Neural Information Processing Systems36(2024)

    Zhuang, Y., Yu, Y., Wang, K., Sun, H., Zhang, C.: ToolQA: A dataset for LLM question answering with external tools. Advances in Neural Information Processing Systems36(2024)

    2024

  25. [33]

    Conference on Empirical Methods in Natural Language Processing (2024)

    Zong, C., Yan, Y., Lu, W., Huang, E., Shao, J., Zhuang, Y.: Triad: A framework leveraging a multi-role LLM-based agent to solve knowledge base question answer- ing. Conference on Empirical Methods in Natural Language Processing (2024)

    2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.