{"id":"362fb973-58f9-47b3-8867-078e29b60ab8","arxiv_id":"2607.18029","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"With an OWL ontology using readable names and annotations, a local Qwen3.6-27B model generates SPARQL for 21 expert questions with 100% accuracy, versus 57% for auto-generated SQL.","lead":"This paper presents NLKGQ, a framework that lets researchers ask natural-language questions over domain metadata by having a local LLM generate SPARQL queries from an OWL ontology. On 21 expert-written questions about a neuroimaging archive, the best setup hit 100% SPARQL accuracy without fine-tuning — but the question set itself was used to iteratively tune the ontology and prompts.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 100% zero-shot accuracy is an in-sample artifact of co-evolving test questions with ontology and prompt rider (§3.3, §6.4); held-out evaluation is required.","rationale":"The reader correctly identified the co-evolution of test questions with ontology and rider as the weakest point. My analysis confirms and sharpens this: §3.3 describes the feedback loop, and §4.5 reveals test cases are saved from successful queries, so the 100% is a development-set number. The paper independently concedes this in §6.4. The proposed held-out test would settle the generalization question directly. Secondary concerns—the fix-retry protocol, small N, post-hoc selection of best configurations—are real but secondary; even a clean 21/21 on an independent set would leave quantitative generality to other domains open, but that is already acknowledged as a limitation. The qualitative finding that ontology naming and annotations dominate is supported by the ablation across eight representations and is valuable independent of the 100% figure. The conditional verdict remains appropriate: the framework and analysis are plausible and useful, but the headline accuracy claim needs a stronger held-out evaluation before it can be accepted as evidence of zero-shot generalization.","tokens_in":14417,"tokens_out":4965,"duration_ms":57373,"concrete_test":"Freeze the released ontology, prompt rider, and generic prompts. Ask domain experts who did not participate in ontology development to write 20–30 new natural-language questions from the same archive, with the constraint that they are not paraphrases of the existing 21. Run each through the harness using Qwen3.6-27B at temperature 0 with the baseline prompt, disable the fix-retry path, and compare exact result sets. If accuracy falls below ~80%, the co-evolution concern is confirmed and the headline should be downgraded to 'in-development-set accuracy'; if it stays near 100%, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim—100% zero-shot SPARQL accuracy—rests on a 21-question competency set that the paper itself describes as co-evolving with the ontology and the prompt rider. Section 3.3 states that test failures feed back into renames and rider directives, and §4.5 says test cases are generated by the 'Save Test Case' function 'directly from a successful query.' Thus the test set and the artifacts being evaluated are jointly optimized: each question is either a query the system already answers or one that has been made answerable by renaming/annotation/rider changes. The 100% therefore measures fit to the development set, not zero-shot generalization to novel phrasings. Section 6.4 concedes 'accuracy on novel end-user queries may differ from test set performance.' In addition, the protocol permits up to two fix-retries on syntax errors (§4.5), so a strict zero-shot reading is not what was actually measured. Without a frozen-ontology, held-out question set, the reported 100% does not establish the abstract's zero-shot claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents NLKGQ, a framework and development process for enabling natural-language access to domain-specific metadata by first capturing domain vocabulary in a well-designed OWL ontology, then using a local LLM to generate SPARQL (or SQL) queries zero-shot. The framework includes a web interface, a query harness, a combinatorial test driver, and an automatic OWL-to-SQL schema generator. The demonstration domain is a neuroimaging archive. The headline result is that the best configuration achieves 100% accuracy on a 21-question competency/regression set for SPARQL, with 57% for auto-generated SQL, and an ablation across eight ontology representations suggests that readable entity names and semantic annotations matter more than model choice or prompt engineering.","tokens_in":14722,"tokens_out":2537,"duration_ms":30147,"significance":"If the central claim is valid—that a small local LLM plus a carefully named and annotated OWL ontology suffices for accurate zero-shot NL-to-SPARQL on domain metadata—this is a genuinely useful engineering contribution. The paper provides a reusable infrastructure, a transparent OWL-to-SQL conversion, a combinatorial evaluation harness, and a practical demonstration on institutional hardware with privacy constraints. The authors also openly acknowledge the co-evolution of test cases with the ontology and the small test-set size. However, the reported 100% accuracy currently measures performance on a development set, not generalization to novel queries; the ablation is confounded by model choice across representations; and the 'zero-shot' label is weakened by the fix-retry protocol. These issues are load-bearing for the abstract's strongest claims, so the current evidence does not yet support the zero-shot generalization statement. The framework and process remain valuable as a reported system, but the evaluation needs to be substantially strengthened before the claims can be accepted.","major_comments":[{"comment":"The 100% accuracy figure is an in-sample measure. §3.3 states that test failures feed back into ontology renames, rider directives, and new test cases; §4.5 says 'Save Test Case' creates artifacts 'directly from a successful query.' Thus the 21 test questions co-evolved with the ontology and prompt, and the reported accuracy is a fit to the development set. The limitation statement in §6.4 concedes that 'accuracy on novel end-user queries may differ from test set performance,' but the abstract and §5.1 still frame the result as zero-shot. A held-out evaluation, with the ontology, rider, and question set frozen before collecting new questions from end users or domain experts, is required to support the headline claim. Without this, the central quantitative claim is not established.","section":"§3.3, §4.5, §6.4"},{"comment":"The 'zero-shot' claim is further weakened by the fix-retry protocol. §3.4 admits that retrying failed queries is 'technically not zero-shot,' and §4.5 permits up to two correction attempts on syntax errors. The paper does not report how many of the 21 successes required retries, nor accuracy without retries. If, for example, only a small fraction of runs needed a second attempt, the deviation may be minor; but as reported, the reader cannot tell whether the 100% reflects the model's first-generation capability or error correction. Please report first-attempt accuracy and the distribution of retries per configuration.","section":"§3.4, §4.5"},{"comment":"The ablation across ontology representations is confounded by model selection. Table 4 reports the 'best configuration for each ontology representation,' but the winning model differs across rows: 'default' uses Q36.27B.D, while 'abstract-dict' uses Q36.35B.M and 'abstract-graph' uses Q36.35B.F. Since Table 3 shows large accuracy differences between models (e.g., 27B dense vs 35B MoE), the accuracy drop from 100% (default) to 19%/10% (abstracts) cannot be attributed solely to the representation. The claim that 'readable names and annotations are the dominant factor' requires holding the model fixed across representations, or at least reporting per-model results. Additionally, 21 questions is small; binning into percentages with no error bars overstates precision. Please provide a model-controlled ablation and, ideally, confidence intervals or per-question breakdowns.","section":"§5.2, Table 4"},{"comment":"The SPARQL-vs-SQL comparison is also affected by the same co-evolved test set and by selecting different best configurations for each backend (SPARQL uses 27B dense at 100%, SQL uses the Q8 model at 57%). While the direction of the gap is plausible and the per-query breakdown in Table 8 is informative, the comparison is not controlled for model, temperature, or prompt. The conclusion that OWL provides a structural advantage over SQL DDL is reasonable as a working hypothesis, but the current data cannot rule out that a different prompt or model would reduce the gap. The authors should temper the conclusion or provide additional runs with the same model on both backends.","section":"§5.4, Table 7"}],"minor_comments":[{"comment":"Typo: 'adevelopment process' should be 'a development process.'","section":"§3"},{"comment":"The paper says '20+ competency questions' in §4.5 but Table 2 lists exactly 21. Please standardize the count.","section":"General"},{"comment":"In the abstract and intro, 'the best model achieves 100%' might be misread as all models. Clarify that this is the best configuration among the 27B dense models, and that other models achieve lower accuracy.","section":"§5.1"},{"comment":"The 'Tokens' column is useful; consider also reporting the number of triples or classes in each representation to help readers interpret the effect of compactness.","section":"Table 4"},{"comment":"The limitation on context window is stated, but the implication for scaling to larger ontologies is only briefly discussed; a more precise bound (e.g., tokens per class/property) would help practitioners.","section":"§6.4"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid engineering report with a clear framework and a transparent—perhaps too transparent—acknowledgment of the evaluation's limits. The co-evolution of test questions with the ontology is a serious threat to the headline zero-shot claim; a held-out evaluation is essential. The ablation's model confound is also fixable within the existing infrastructure. I would not reject: the framework and process are valuable, and the authors seem capable of running the additional experiments needed. However, as it stands, the paper overstates what the data support. The recommendation is major_revision, largely to enable a controlled, held-out evaluation and a model-controlled ablation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know up front. The real contribution is the ontology-first recipe and the controlled OWL→SQL comparison; the headline 100% SPARQL accuracy should not be quoted as zero-shot evidence. The stress-test note lands: §3.3 and §6.4 explicitly say the competency questions, ontology, and prompt rider co-evolved, and §4.5 permits up to two fix-retries on syntax errors. So the 100% is a measure of iterative fit to a 21-question development set, not generalization to novel phrasings. The authors do concede this in §6.4, but the abstract and results sections overstate it.\n\nWhat is genuinely new and useful: the end-to-end process description, the reusable test driver and web framework, the 8-representation ablation, and the automatic OWL-to-SQL schema generation that allows a same-data, same-question SPARQL-vs-SQL comparison. The qualitative findings are credible and align with prior work: readable names and annotations are the dominant factors, quantization preserves accuracy, and dense 27B beats larger MoE models for this structured generation task. The ablation across ontology representations, despite its flaws, is the most valuable part of the paper.\n\nSoft spots, in proportion. The reported numbers are best-configuration maxima over 768 configurations, with no error bars or repetition variance. The per-representation ablation is confounded because the best row for each representation sometimes uses a different model — default uses the 27B dense, abstract representations use 35B MoE — so the representation effect is mixed with model choice. The test set is small, and the “Save Test Case” function creates reference cases from successful queries, which makes the circularity concrete. The SQL comparison is interesting but only against an auto-generated wide-table schema; it does not show OWL beats a hand-designed relational schema, only that an ontology-authored schema beats an auto-generated DDL. The EAV schema result is too thinly reported to support strong conclusions.\n\nThe citation pattern is fine, and the paper is honest about the relevant prior work. It is a practical systems/engineering paper, not a benchmark contribution. For ontology engineers and KGQA researchers, the design principles and the iterative process are worth engaging with.\n\nRecommendation: send to peer review, but require either a genuinely held-out question set with frozen ontology and rider and no retries, or a reframed claim that the 100% is development-set accuracy. The framework and ablation deserve referee attention; the headline needs to stop overstating what was measured.","headline":"A genuinely useful ontology-first recipe with a clean SPARQL-vs-SQL comparison, but the headline 100% accuracy is development-set fit — the 21 questions co-evolved with the ontology and rider.","tokens_in":15184,"tokens_out":2072,"would_cite":true,"duration_ms":25345,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An LLM can turn natural-language questions into perfectly accurate SPARQL queries when the domain ontology is designed to be readable and semantically annotated, hitting 100% on a 21-question expert-built test set in a neuroimaging archive.","keywords":["natural language query","SPARQL","SQL","OWL ontology","zero-shot LLM","metadata search","knowledge graph","neuroimaging metadata"],"falsifier":"Collect a fresh set of natural-language metadata questions from actual end users after the ontology, prompt rider, and competency set are frozen; run the same best model and full Turtle ontology on this held-out set with no further edits. If accuracy falls well below 100% (or below the regression-set level), the claimed zero-shot capability is an artifact of co-design rather than a property of the ontology representation alone.","tokens_in":14275,"feed_emoji":"🧠","tokens_out":3116,"duration_ms":36741,"temperature":0.7,"pith_summary":"The paper claims that the bottleneck in natural-language access to domain metadata is not the LLM but the ontology: if domain vocabulary and semantics are captured in a well-designed OWL ontology with full English names and rich annotations, a locally deployed LLM can generate correct SPARQL queries zero-shot, without fine-tuning, retrieval, or multi-agent orchestration. On a 21-question competency set built with domain experts, the best configuration reaches 100% SPARQL accuracy, while an auto-generated SQL backend from the same ontology reaches only 57%. An ablation across eight ontology representations shows that readable entity names and semantic annotations are the dominant accuracy factors, more so than model choice or prompting. The demonstration domain is a large neuroimaging archive where privacy rules force local deployment on modest hardware, and a quantized 27B dense model achieves the same perfect accuracy as a full-precision model.","feed_headline":"Ontology design lifts LLM query accuracy to 100 percent","feed_subtitle":"Readable names and annotations beat model size and prompts; SQL on the same data reaches only 57 percent.","key_machinery":"The central mechanism is the OWL ontology itself, treated as a single source of truth and consumed directly by the LLM in its full Turtle form. Design principles (full English words, has-Property naming, explicit domain and range, rdfs:comment and skos:altLabel annotations, no opaque codes) make the schema self-describing. The ETL pipeline materializes implied triples so no reasoner is needed at query time, and a prompt rider encodes conventions that naming cannot express. The ablation across eight representations isolates exactly which parts of this machinery matter: names and annotations dominate, while prompt formulation and temperature have only minor effects.","core_discovery":"The paper establishes that when an OWL ontology is deliberately engineered with full-word property names, consistent naming patterns, explicit domain/range, natural-language comments, labels, and synonyms, a local LLM can generate correct SPARQL for unseen natural-language questions with no training examples. The full Turtle ontology is placed in the system prompt; the LLM uses its names and annotations to map the question to graph patterns, inverse properties, and filters. The strongest measure is 100% accuracy on 21 competency/regression questions covering lookups, joins, filters, aggregations, and ontology introspection. The paper further shows that stripping comments and labels drops SPA","pith_inferences":["Because the 21 competency questions co-evolved with the ontology and rider, the 100% figure likely measures the tuning loop rather than true zero-shot generalization; a held-out set of questions frozen after development would be a stronger test and might score notably lower.","The large SPARQL-vs-SQL gap hints that relational schemas could be redesigned with ontology-like features—readable names, explicit join directions, and rich column comments—to narrow the gap; this is a testable design hypothesis the paper does not pursue.","The reported MoE underperformance is from one model family and one task; readers should not generalize to all mixture-of-experts architectures without broader evaluation.","For ontologies too large to fit the context window, the paper's token-dense compact representations lose ~30 points of accuracy, so future work will likely need hierarchical or summarized ontology encodings to scale the method."],"forward_implications":["Institutions with privacy constraints can deploy a local ~27B dense quantized model and achieve complete SPARQL query accuracy on curated questions, without outsourcing data to external APIs.","For new domain metadata projects, the paper implies that investing in ontology naming and annotation quality is more effective than investing in fine-tuning, retrieval infrastructure, or complex prompt engineering.","When a knowledge graph is available, SPARQL is likely a better target backend than auto-generated SQL for LLM-based natural-language access, because OWL's structural features transfer to the prompt more directly.","The co-evolution process (ontology, competency questions, and prompt rider iterating together) makes accuracy measurable and regression-free, providing a reusable development loop for other domains.","The result suggests that compact or abstract ontology representations, which fit smaller contexts, come at a substantial accuracy cost, so fitting the full annotated ontology in context should be the default when possible."],"fun_headline_variants":["Ontology naming and comments beat model size for LLM SPARQL","Readable ontology annotations key to zero-shot SPARQL accuracy","Well-designed OWL ontology lifts LLM query generation to 100%","Domain ontology, not prompt tweaks, drives LLM to perfect SPARQL","Descriptive ontology terms make LLM query generation 100% accurate"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the 21 competency questions are representative of real user questions and were not effectively optimized into the ontology and prompt rider during the iterative development cycle; if novel end-user queries differ substantially, the 100% accuracy may not carry over.","fun_headline_variants_meta":{"raw":{"variants":["Ontology naming and comments beat model size for LLM SPARQL","Readable ontology annotations key to zero-shot SPARQL accuracy","Well-designed OWL ontology lifts LLM query generation to 100%","Domain ontology, not prompt tweaks, drives LLM to perfect SPARQL","Descriptive ontology terms make LLM query generation 100% accurate"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000302,"raw_usage":{"total_tokens":1617,"prompt_tokens":823,"completion_tokens":794,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":567,"completion_tokens_details":{"reasoning_tokens":713}},"tokens_in":567,"tokens_out":794,"duration_ms":8374,"temperature":1.0,"reasoning_tokens":713,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T16:18:47.635068+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Collect a fresh set of natural-language metadata questions from actual end users after the ontology, prompt rider, and competency set are frozen; run the same best model and full Turtle ontology on this held-out set with no further edits. If accuracy falls well below 100% (or below the regression-set level), the claimed zero-shot capability is an artifact of co-design rather than a property of the ontology representation alone.","supporting_citations":[],"review_version":1}