REVIEW 4 major objections 6 minor 36 references
Automating SPARQL Query Translations between DBpedia and Wikidata
T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read This paper sets out to show that a large open language model, given an explicit schema-mapping table and a few worked examples, can translate SPARQL queries between different knowledge graphs, reporting 86% accuracy on two test pairs.
desk verdict A useful first benchmark for LLM-based SPARQL translation, with headline accuracy plausible as an upper bound; needs an uncurated test set and multi-run variance before the scalability claim is trusted. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The carrying mechanism is the prompt's 'er2' variable: a JSON mapping table pairing each source-graph IRI with its target-graph IRI, for instance mapping DBpedia's dbo:director to Wikidata's wdt:P57. For DBpedia and Wikidata the table is built automatically from owl:sameAs, owl:equivalentProperty, and owl:equivalentClass links; for DBLP and OpenAlex it is built from shared ORCID identifiers plus manually aligned relations. The prompt combines the natural-language question, the source SPARQL query, the two graph names, this mapping table, and an instruction to emit the final query between <sparql> tags. Four few-shot examples, chosen with sentence embeddings and clustering for diversity, are added in the strongest configuration. The mapping table is what does the work: removing it drops accuracy to near zero, while adding it lifts even the smallest model by tens of percentage points.
What would settle it
Take the full set of source queries before filtering—including those that fail on one of the two graphs or return empty results—translate each with the best configuration, and execute the outputs against stable snapshots. If accuracy on this unfiltered set falls well below 86%, or if the excluded queries concentrate the failures, then the reported accuracy is a property of the curated benchmark rather than of the general translation pathway.
Extended reading notes
Core claim
On its own terms, the paper discovers that SPARQL-to-SPARQL translation is not a generic reasoning task the models solve from their pretrained knowledge; it is a lookup-plus-restructure task that succeeds when the prompt tells the model which source IRI corresponds to which target IRI. With that mapping supplied, the largest model translates 86 of 100 Wikidata queries into DBpedia queries whose executed answer sets exactly match the gold answers, and the same model translates 86 of 100 DBLP queries to OpenAlex when four examples are included. Without mappings or examples, accuracy collapses—one model scores 0% on DBpedia-to-Wikidata and 1% on DBLP-to-OpenAlex. The paper also finds that errors are dominated by structural mistakes that often cascade from a single wrong mapping, and that the simpler human-readable identifiers of DBpedia make it an easier target than Wikidata's numeric identifiers.
Load-bearing premise
The benchmarks contain only queries whose gold SPARQL already works on both graphs and returns non-empty comparable answers, so the reported accuracy is measured on a filtered, well-behaved slice of real translation demand rather than on all queries a user might want to move between these knowledge graphs.
Editorial extensions
If this is right
- A practical recipe for cross-graph query translation is: use a large open LLM, supply a JSON entity-relation mapping table, and include four diverse examples in the prompt; no fine-tuning or per-graph training is required.
- The same recipe transfers from encyclopaedic graphs to a specialised scholarly pair, so it is plausible for other graph pairs with comparable mappings, not just DBpedia and Wikidata.
- Translation is not symmetric: Wikidata-to-DBpedia is consistently easier than DBpedia-to-Wikidata, so evaluation and tooling should treat direction as a first-class factor.
- The strict exact-answer-set evaluation means the reported accuracy is a conservative measure: a translation that returns the same answers in different formatting or with extra variables would be marked incorrect even if semantically faithful.
- Keeping the mapping table up to date becomes an ongoing requirement, since graph schemas evolve and stale mappings will produce the structural errors the paper catalogues.
Reading between the lines
- Because both benchmarks were pre-filtered to queries that execute successfully on the source and target graphs, the 86% figures describe an upper bound for curated, translatable queries; real workloads containing graph-specific constructs, failing queries, or deep domain expertise would likely score lower.
- The method could be used to bootstrap mappings for a new graph pair: generate candidate mappings with an LLM, verify or correct a handful by hand, then use the corrected set as the mapping table for larger-scale translation.
- The error taxonomy points to a self-correction loop: since a single wrong mapping often cascades into a structural error, running the generated query, comparing the error type, and feeding that diagnosis back into the prompt may recover many failures without retraining.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper investigates whether open-weight LLMs can translate SPARQL queries between heterogeneous knowledge graphs, focusing on DBpedia↔Wikidata and generalizing to DBLP→OpenAlex. The authors construct two 100-query benchmarks: one derived from QALD-9-Plus training queries that execute successfully on both DBpedia and Wikidata and return comparable non-empty results, and one derived from DBLP-QuAD templates deemed semantically translatable to OpenAlex, with manually authored gold OpenAlex queries. They evaluate Llama-3.1-8B, DeepSeek-R1-Distill-Llama-70B, and Mistral-Large-Instruct-2407 under zero-shot, few-shot, and chain-of-thought prompting, with and without explicit entity-relation mapping tables. The headline results are 86% accuracy for Mistral-Large-Instruct-2407 on Wikidata→DBpedia (few-shot) and 86% on DBLP→OpenAlex (few-shot), with markedly lower performance in the opposite direction and for smaller models. The paper also contributes a detailed error taxonomy and analyzes co-occurrence of error types. The central claim is that LLMs with structured prompting and explicit schema mappings offer a viable and scalable pathway toward KG interoperability.
Significance. If the reported results are representative of operational translation demand, this is a meaningful step toward automating cross-KG SPARQL translation, a relatively underexplored task. The paper is, to the best of my knowledge, the first systematic study of SPARQL-to-SPARQL translation with LLMs on multiple KG pairs. Strengths include the public release of code and data, the use of stable local KG snapshots for reproducibility, the inclusion of a generalization task beyond encyclopaedic KGs, and a structured error analysis that goes beyond simple accuracy reporting. The main caveat is that the benchmarks are curated in ways that may remove exactly the hard and realistic cases, so the headline accuracies are better interpreted as upper bounds on performance for uncurated real-world queries. This limits the strength of the scalability claim in the abstract, but the empirical methodology is otherwise careful and the findings are informative for future work.
major comments (4)
- [§3.1, Table 1] The primary benchmark is constructed by keeping only QALD-9-Plus training queries that 'successfully executed on both Wikidata and DBpedia and returned non-empty, comparable results,' reducing 371 queries to a curated 100. This filter removes precisely the queries that would be hardest to translate in practice: those that fail on one graph, use graph-specific constructs, or return empty or non-comparable results. Consequently, the 86% accuracy reported in §5.1 is an upper bound on accuracy for an uncurated set of real-world translation requests, and the abstract's claim of a 'viable and scalable pathway toward KG interoperability' is stronger than the evidence supports. I request that the authors quantify how many of the 371 queries were excluded for each reason, present results on the full set if feasible, or explicitly reframe the claims as applying to the curated, executable subset.
- [§3.2, §5.3] The DBLP→OpenAlex benchmark was created by selecting 100 DBLP-QuAD query templates 'semantically translatable to OpenAlex' and manually creating the gold OpenAlex SPARQL queries. This introduces two sources of optimistic bias: queries for DBLP-specific constructs (e.g., BibTeX types) are excluded, and the gold standard is the authors' own manual translation, so the exact-match evaluation in §3.4 may penalize equally valid alternative translations that an independent expert would accept. The paper should report how many DBLP-QuAD templates were excluded and why, and ideally have the gold queries validated by a second annotator or compare a sample of LLM 'incorrect' outputs against an independent human judgement of equivalence.
- [§5.1, §5.3, §4.2] All accuracy numbers are single-run point estimates. Section 7 notes the stochasticity of LLM outputs, but no variance is reported. For N=100, the standard error of an 86% accuracy is approximately 3.4 percentage points, and different sampling of the one run could produce materially different results. This is load-bearing because the paper's main contribution is the 86% figures, not a qualitative phenomenon. I recommend running each configuration at least three times with different random seeds (or temperature settings) and reporting mean and standard deviation, particularly for the Mistral-Large-Instruct-2407 few-shot results.
- [§3.4] The evaluation uses a strict exact-match criterion: an LLM-generated query is correct only if its executed result set precisely matches the gold answer set. This criterion can both under- and over-estimate translation quality. It underestimates when a translation is semantically correct but uses different variable names, different but equivalent triple patterns, or different FILTER formulations that yield the same results; it overestimates if the gold query itself is not the only reasonable interpretation of the natural language question. Given the DBLP gold queries are manually authored by the authors, I suggest supplementing exact-match with a secondary evaluation, such as manual inspection of a random sample of 'incorrect' translations to estimate the false-negative rate, or a relaxed equivalence criterion based on answer-set overlap.
minor comments (6)
- [§3.3] The mapping process left 144 of 637 DBpedia terms unmapped; the paper says queries with unmapped terms were retained to test robustness, but it is not reported how many of the 100 benchmark queries contain unmapped terms and whether these queries correlate with lower accuracy. Adding such an analysis would strengthen the discussion of ER mapping coverage.
- [§4.2.2] The zero-shot baseline without ER mapping was applied only to Llama 3.1-8B and Mistral-Large-Instruct-2407, not to DeepSeek-R1-Distill-Llama-70B; clarify whether this was an intentional experimental design choice and whether it affects the comparability of model results.
- [§5.1, Figures 2–3] The figures are referenced but not shown in the text excerpt provided; the captions say 'Correctly Translated Queries' but do not indicate error bars or statistical significance. If error bars are not available, state this explicitly in the captions.
- [§3.5, Table 2] The NLQ categorization is useful, but it is unclear whether the distribution of the 100 selected queries matches the original QALD-9-Plus train set. A brief comparison of category proportions would help assess representativeness.
- [§4.2.1] The variable name 'er2' for the entity-relation mapping is not self-explanatory and appears without introduction; consider renaming it to 'schema_mapping' or a similar descriptive term throughout the paper and the provided JSON example.
- [§5.2, Table 3] The error counts are aggregated over 12 configurations per direction, which conflates model and prompting effects. Reporting per-configuration error distributions in an appendix would make the error analysis more actionable.
Circularity Check
No circular derivation; empirical evaluation on earlier external benchmark datasets, with only minor non-load-bearing self-citation.
full rationale
The paper makes no formal derivation; its central claims are empirical accuracies on curated benchmarks. The evaluations compare LLM-generated SPARQL translations against gold answers obtained by executing the original QALD-9-Plus queries on static snapshots (Section 3.1) or by manually authored OpenAlex gold queries (Section 3.2). Neither gold standard is defined in terms of the LLM outputs, so there is no self-definitional or fitted-input-called-prediction step. The benchmark sources, QALD-9-Plus [15] and DBLP-QuAD [27], are co-authored by current paper authors, but they are earlier published datasets that do not depend on the present results, and few-shot examples are explicitly selected from non-test data with no overlap (Section 4.2.2), so the 86% figure is not forced by construction. The Section 3.1 filter that keeps only queries executing successfully on both KGs with non-empty comparable results is a benchmark representativeness limitation, acknowledged in Section 7, rather than a circularity. No uniqueness theorem or load-bearing argument is imported from self-citation. The only notable self-citation is the reuse of the authors' own datasets and prior error-category work [29], which does not affect the derivation chain. Score is therefore low; the paper is an empirical study with external benchmarks and no reduction of the claimed predictions to their inputs.
Assumptions & free parameters
free parameters (1)
- few_shot_example_count =
4
assumptions (5)
- domain assumption Exact match of executed result sets, with order ignored unless inherently meaningful, is a sufficient correctness criterion for translated SPARQL queries.
- domain assumption QALD-9-Plus provides correct gold SPARQL queries for both DBpedia and Wikidata.
- domain assumption The manually created OpenAlex gold queries correctly capture the intent of the selected DBLP queries.
- domain assumption Single LLM runs are sufficient to estimate translation accuracy despite acknowledged stochasticity.
- domain assumption The curated 100-query subsets are representative of real-world SPARQL translation demand.
Cite this review
Pith. "Pith review of Automating SPARQL Query Translations between DBpedia and Wikidata." pith.science (2026). https://pith.science/paper/VSRVKF6C
@misc{pith2026250710045,
author = {Pith},
title = {Pith review of: Automating SPARQL Query Translations between DBpedia and Wikidata},
year = {2026},
howpublished = {\url{https://pith.science/paper/VSRVKF6C}},
note = {Machine review of arXiv:2507.10045}
}
read the original abstract
This paper investigates whether state-of-the-art Large Language Models (LLMs) can automatically translate SPARQL between popular Knowledge Graph (KG) schemas. We focus on translations between the DBpedia and Wikidata KG, and later on DBLP and OpenAlex KG. This study addresses a notable gap in KG interoperability research by rigorously evaluating LLM performance on SPARQL-to-SPARQL translation. Two benchmarks are assembled, where the first align 100 DBpedia-Wikidata queries from QALD-9-Plus; the second contains 100 DBLP queries aligned to OpenAlex, testing generalizability beyond encyclopaedic KGs. Three open LLMs: Llama-3-8B, DeepSeek-R1-Distill-Llama-70B, and Mistral-Large-Instruct-2407 are selected based on their sizes and architectures and tested with zero-shot, few-shot, and two chain-of-thought variants. Outputs were compared with gold answers, and resulting errors were categorized. We find that the performance varies markedly across models and prompting strategies, and that translations for Wikidata to DBpedia work far better than translations for DBpedia to Wikidata.
Figures
Reference graph
Works this paper leans on
-
[1]
Vrande ˇci´c D, Krötzsch M. Wikidata. Communications of the ACM. 2014;57(10):78-85. doi:10.1145/2629489
doi:10.1145/2629489 2014
-
[2]
DB- pedia – A large-scale, multilingual knowledge base extracted from Wikipedia
Lehmann J, Isele R, Jakob M, Jentzsch A, Kontokostas D, Mendes PN, et al. DB- pedia – A large-scale, multilingual knowledge base extracted from Wikipedia. Se- mantic Web. 2015;6(2):167-95. doi:10.3233/SW-140134
-
[3]
A comprehensive survey of entity alignment for knowledge graphs
Zeng K, Li C, Hou L, Li J, Feng L. A comprehensive survey of entity alignment for knowledge graphs. AI Open. 2021;2:1-13. doi:10.1016/j.aiopen.2021.02.002
-
[4]
RDF 1.1 Primer [W3C Recommendation]; 2014
Schreiber G, Raimond Y , Manola F, Miller E, McBride B. RDF 1.1 Primer [W3C Recommendation]; 2014. Accessed: February 5, 2025. Available from:https: //www.w3.org/TR/rdf11-primer/
work page 2014
-
[5]
Semantics and complexity of SPARQL
Pérez J, Arenas M, Gutierrez C. Semantics and complexity of SPARQL. ACM Trans Database Syst. 2009;34(3):16:1-16:45. doi:10.1145/1567274.1567278
arXiv 2009
-
[6]
A survey of RDF stores & SPARQL engines for querying knowledge graphs
Ali W, Saleem M, Yao B, Hogan A, Ngomo ACN. A survey of RDF stores & SPARQL engines for querying knowledge graphs. The VLDB Journal. 2022;31(3):1-26. doi:10.1007/s0 0778-021-00711-3
doi:10.1007/s0 2022
-
[7]
Hogan A, Cochez M, de Melo G. Knowledge graphs. vol. 22 of Synthesis lectures on data, semantics and knowledge. Cham: Springer; 2022
work page 2022
-
[8]
Construction of Knowledge Graphs: Current State and Challenges
Hofer M, Obraczka D, Saeedi A, Köpcke H, Rahm E. Construction of Knowledge Graphs: Current State and Challenges. Information. 2024;15(8):509. doi:10.3390/info15080509
Show all 36 references
-
[9]
Atten- tion is All you Need
Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, et al. Atten- tion is All you Need. In: Proceedings of the 31st International Conference on Neu- ral Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA; 2017. p. 6000-10. Available from:h...
2017
-
[10]
Language Models are Few-Shot Learners
Brown TB, Mann B, Ryder N, Subbiah M, Kaplan J, Dhariwal P, et al. Language Models are Few-Shot Learners. In: Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12; 2020. p. 1877-901
2020
-
[11]
Neural machine translating from nat- ural language to SPARQL
Yin X, Gromann D, Rudolph S. Neural machine translating from nat- ural language to SPARQL. Future Gener Comput Syst. 2021;117:510-9. doi:10.1016/j.future.2020.12.013
2021 doi
-
[12]
Can Knowledge Graphs Reduce Hal- lucinations in LLMs? : A Survey
Agrawal G, Kumarage T, Alghamdi Z, Liu H. Can Knowledge Graphs Reduce Hal- lucinations in LLMs? : A Survey. In: Duh K, Gomez H, Bethard S, editors. Pro- ceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langua...
2024 doi
-
[13]
Unifying Large Language Models and Knowledge Graphs: A Roadmap
Pan S, Luo L, Wang Y , Chen C, Wang J, Wu X. Unifying Large Language Models and Knowledge Graphs: A Roadmap. IEEE Transactions on Knowledge and Data Engineering. 2024;36(7):3580-99. doi:10.1109/TKDE.2024.3352100
2024
-
[14]
A Survey of Knowl- edge Graph Reasoning on Graph Types: Static, Dynamic, and Multi-Modal
Liang K, Meng L, Liu M, Liu Y , Tu W, Wang S, et al. A Survey of Knowl- edge Graph Reasoning on Graph Types: Static, Dynamic, and Multi-Modal. IEEE transactions on pattern analysis and machine intelligence. 2024;46(12):9456-78. doi:10.1109/TPAMI.2024.3417451. July 2025
2024
-
[15]
QALD-9-plus: A Multilingual Dataset for Question Answering over DBpedia and Wikidata Translated by Na- tive Speakers
Perevalov A, Diefenbach D, Usbeck R, Both A. QALD-9-plus: A Multilingual Dataset for Question Answering over DBpedia and Wikidata Translated by Na- tive Speakers. In: 16th IEEE International Conference on Semantic Comput- ing, ICSC 2022, Laguna Hills, CA, USA, January 26-28. I...
2022
-
[16]
The DBLP Computer Science Bibliography: Evolution, Research Issues, Perspectives
Ley M. The DBLP Computer Science Bibliography: Evolution, Research Issues, Perspectives. In: Laender AHF, Oliveira AL, editors. String Processing and Infor- mation Retrieval. SpringerLink Bücher. Berlin, Heidelberg: Springer-Verlag Berlin Heidelberg; 2002. p. 1-10. doi:10.1007...
2002 doi
- [17]
-
[18]
KG-Hub- building and exchanging biological knowledge graphs
Caufield JH, Putman T, Schaper K, Unni DR, Hegde H, Callahan TJ, et al. KG-Hub- building and exchanging biological knowledge graphs. Bioinformatics (Oxford, England). 2023;39(7). doi:10.1093/bioinformatics/btad418
2023 doi
-
[19]
A survey on augmenting knowl- edge graphs (KGs) with large language models (LLMs): models, evaluation met- rics, benchmarks, and challenges
Ibrahim N, Aboulela S, Ibrahim AF, Kashef RF. A survey on augmenting knowl- edge graphs (KGs) with large language models (LLMs): models, evaluation met- rics, benchmarks, and challenges. Discov Artif Intell. 2024;4(1):76. doi:10.1007/s 44163-024-00175-8
2024 doi
-
[20]
Knowledge graph augmentation: consistency, immutability, reliability, and context
Takan S. Knowledge graph augmentation: consistency, immutability, reliability, and context. PeerJ Comput Sci. 2023;9:e1542. doi:10.7717/peerj-cs.1542
2023 doi
-
[21]
Querying and Searching Heterogeneous Knowl- edge Graphs in Real-time Linked Dataspaces
Freitas A, O’Riain S, Curry E. Querying and Searching Heterogeneous Knowl- edge Graphs in Real-time Linked Dataspaces. In: Real-time Linked Dataspaces - Enabling Data Ecosystems for Intelligent Systems. Springer; 2020. p. 105-24. doi:10.1007/978-3-030-29665-0_7
2020 doi
-
[22]
Knowledge Graphs Querying
Khan A. Knowledge Graphs Querying. SIGMOD Rec. 2023;52(2):18-29. doi:10.1145/3615952.3615956
2023
-
[23]
A Polystore Architecture Using Knowledge Graphs to Support Queries on Heterogeneous Data Stores
Azevedo LG, Souza RFS, Soares, Elton Figueiredo de Souza, Thiago RM, Tesolin JCC, Oliveira AC, et al. A Polystore Architecture Using Knowledge Graphs to Support Queries on Heterogeneous Data Stores. CoRR. 2023;abs/2308.03584. doi:10.48550/arXiv.2308.03584
-
[24]
Knowledge Graphs: A Practical Review of the Research Landscape
Kejriwal M. Knowledge Graphs: A Practical Review of the Research Landscape. Inf. 2022;13(4):161. doi:10.3390/info13040161
2022 doi
-
[25]
Semantics preserving SPARQL-to-SQL translation
Chebotko A, Lu S, Fotouhi F. Semantics preserving SPARQL-to-SQL translation. Data Knowl Eng. 2009;68(10):973-1000. doi:10.1016/j.datak.2009.04.001
2009 doi
-
[26]
Sorry, i don’t speak SPARQL: translating SPARQL queries into natural language
Ngomo ACN, Bühmann L, Unger C, Lehmann J, Gerber D. Sorry, i don’t speak SPARQL: translating SPARQL queries into natural language. In: 22nd International World Wide Web Conference, WWW ’13, Rio de Janeiro, Brazil, May 13-17, 2013
2013
-
[27]
DBLP-QuAD: A Question Answering Dataset over the DBLP Scholarly Knowledge Graph
Banerjee D, Awale S, Usbeck R, Biemann C. DBLP-QuAD: A Question Answering Dataset over the DBLP Scholarly Knowledge Graph. In: Proceedings of the 13th International Workshop on Bibliometric-enhanced Information Retrieval co-located with 45th European Conference on Information ...
2023
-
[28]
Farewell Freebase: Migrating the SimpleQues- tions Dataset to DBpedia
Azmy M, Shi P, Lin J, Ilyas IF. Farewell Freebase: Migrating the SimpleQues- tions Dataset to DBpedia. In: Proceedings of the 27th International Conference on Computational Linguistics, COLING 2018, Santa Fe, New Mexico, USA, August 20-26; 2018. p. 2093-103. Available from:htt...
2018
-
[29]
Modern Baselines for SPARQL Semantic Parsing
Banerjee D, Nair PA, Kaur JN, Usbeck R, Biemann C. Modern Baselines for SPARQL Semantic Parsing. In: SIGIR ’22: The 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, Madrid, Spain, July 11 - 15; 2022. p. 2260-5. doi:10.1145/3477495.3531841
2022
- [30]
-
[31]
Mistral Large 2
Mistral AI. Mistral Large 2. Mistral AI; 2024. Accessed: May 5, 2025. Mistral AI Blog Post. Available from:https://mistral.ai/news/ mistral-large-2407
2024
- [32]
-
[33]
Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
Reimers N, Gurevych I. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong ...
2019
-
[34]
Chain- of-Thought Prompting Elicits Reasoning in Large Language Models
Wei J, Wang X, Schuurmans D, Bosma M, ichter b, Xia F, et al. Chain- of-Thought Prompting Elicits Reasoning in Large Language Models. In: Koyejo S, Mohamed S, Agarwal A, Belgrave D, Cho K, Oh A, edi- tors. Advances in Neural Information Processing Systems. vol. 35. Cur- ran As...
2022
-
[2013]
p. 977-88. doi:10.1145/2488388.2488473
-
[2019]
p. 3980-90. doi:10.18653/v1/D19-1410
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.