{"id":"136923d1-3c68-4959-aab2-03cbd4ba467c","arxiv_id":"2502.03992","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"OntoSCPrompt separates SPARQL structure prediction from KG-specific content filling with ontology-guided hybrid prompt learning, achieving competitive benchmark scores while its cross-KG generalization evidence lacks baseline comparisons.","lead":"OntoSCPrompt is a two-stage LLM system that first writes a generic SPARQL query skeleton and then fills it with knowledge-graph-specific names, using ontology text and trainable prompt vectors to transfer across KGs. The paper reports competitive scores on common KGQA benchmarks, but the evidence for its headline cross-KG generalization claim is thinner than the abstract implies.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The cross-KG evidence in Table VI is ambiguous about target-data fine-tuning, so the central 'without retraining' generalization claim is not yet established.","rationale":"The reader's conditional verdict is appropriate, and I agree that the weakest point is the zero-shot generalization protocol. My concern sharpens this to one specific, checkable ambiguity: Table VI does not state whether the 'pre' variant was fine-tuned on the target training splits. The within-dataset results in Table IV are coherent and competitive, and the two-stage architecture plus constrained decoding is a reasonable design, but the central claim of the paper is transfer without retraining. That claim rests entirely on Table VI, since Table V is within the same KG and Table IV uses per-dataset training. The phrase 'pre-trained variant using LC-QuAD 1.0' combined with the PT+FT configuration in Table VII leaves open the possibility that the cross-KG gains come from supervised target fine-tuning rather than ontology-guided zero-shot transfer. The ambiguity is resolvable from code and training logs, so I would not reject the paper; I would keep it conditional on the authors clarifying the protocol and adding a zero-shot baseline comparison. This does not change the reader's verdict, hence UNCHANGED.","tokens_in":13930,"tokens_out":5207,"duration_ms":51219,"concrete_test":"Inspect the released repository (github.com/LongquanJiang/OntoSCPrompt) to determine whether DBLP-QuAD train (7,000) and CoyPu train (873) examples were included in the optimizer updates for the 'pre' variant; then rerun Table VI from an LC-QuAD-1.0-only checkpoint with those target splits withheld and compare F1 on the DBLP-QuAD test set and the 66-question CoyPu test set against a trivial template-frequency baseline. If the withheld-target scores differ materially from 84.6/83.3 or do not beat the baseline, the zero-shot generalization claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section IV.A.2 presents the only evidence for the paper's headline claim of generalizing to unseen KGs: Table VI reports F1 of 78.2/80.2 on DBLP-QuAD/CoyPuKGQA and 84.6/83.3 for OntoSCPromptpre, with 'pre' described only as 'using LC-QuAD 1.0.' In contrast to Table V, where the within-KG zero-shot protocol explicitly states 'without any fine-tuning, only with their ontology provided,' the cross-KG subsection never states whether the DBLP-QuAD (7,000) or CoyPu (873) training splits were used to update parameters. If the pre variant is initialized from LC-QuAD and then fine-tuned on the target training splits, the +6.4/+3.1 margins are supervised adaptation, not zero-shot transfer. That would leave no experiment supporting 'without retraining': Table IV is trained per dataset, and Table V is within-KG rather than across-KG. Because the paper's contribution is precisely a single system that transfers with only ontology and retrieved subgraphs, this ambiguity is load-bearing.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents OntoSCPrompt, a two-stage KGQA system that first predicts a generic SPARQL query structure with placeholders and then fills the placeholders with KG-specific identifiers. The method augments this two-stage framework with ontology-guided hybrid prompts (textual ontology plus learnable continuous vectors) and with task-specific constrained decoding strategies. The authors evaluate the system on WebQSP, CWQ, LC-QuAD 1.0, SimpleDBpediaQA, DBLP-QuAD, and CoyPuKGQA, reporting within-dataset results, zero-shot transfer between datasets on the same KG, and cross-KG results on DBLP-QuAD and CoyPuKGQA. The paper's central claim is that the system can match state-of-the-art KGQA systems without retraining and can generalize to unseen KGs using only the target ontology and retrieved subgraphs. The limitations section explicitly acknowledges remaining challenges, including relation directionality, SPARQL annotation style, and verbose Freebase naming conventions.","tokens_in":14146,"tokens_out":7106,"duration_ms":67671,"significance":"If the cross-KG generalization claim is substantiated, OntoSCPrompt would be a practically valuable contribution: a single system that can answer questions over a new knowledge graph without target-specific training data would reduce the cost of deploying KGQA in new domains. The two-stage structure/content separation is well motivated, the ontology-guided prompt design is coherent, and the constrained decoding strategies address a real failure mode in SPARQL generation. The paper also provides a public code repository, which is a strength for reproducibility. The main significance, however, rests on the cross-KG experiments, and those experiments are currently under-specified in a way that directly affects the headline claim.","major_comments":[{"comment":"The cross-KG experiment is the only direct evidence for the headline claim of generalizing to unseen KGs, but the training protocol is not specified. The text says the 'pre-trained' variant uses LC-QuAD 1.0, and that the model is 'adapted' to a target dataset; it never states whether the DBLP-QuAD (7,000) or CoyPuKGQA (873) training splits were used to update parameters. If those splits were used, the +6.4 and +3.1 gains are supervised adaptation, not zero-shot transfer, and no experiment in the paper supports the abstract's 'without retraining' claim. Please state explicitly whether target training data were used, and report the exact fine-tuning protocol, including epochs and learning rates.","section":"IV.A.2, Table VI"},{"comment":"The ontology prompt in Section II.C(a) is exemplified as 'ontology: concepts: Company, Person; relations: foundedBy, birthDate, deathDate, type; entities: Steve Jobs, Steve Wozniak, Apple Inc.', which includes A-Box entity identifiers, while Section I defines ontology/schema/T-Box as concepts and relations only. Please clarify what information is supplied to the model for an unseen KG: if target entities come from a fixed entity inventory, the wording 'only with their ontology provided' in Section IV.A.1 is misleading; if they come from subgraph retrieval or entity linking, that mechanism should be described, since it is load-bearing for the zero-shot claim.","section":"II.C(a), I (footnote 2)"},{"comment":"The zero-shot within-KG results (48.8 Hits@1 on CWQ, 34.0 F1 on SimpleDBpediaQA) are presented without any zero-shot baseline. The sentence 'From earlier papers, we know that other models do not achieve any hits [35]' cites Hartmann et al., which is a dataset-generation paper, not a cross-dataset zero-shot study. Please either provide a direct comparison of existing systems under the same zero-shot protocol or remove the unsupported assertion; as written, the reader cannot judge whether 48.8/34.0 is a meaningful degree of generalization.","section":"IV.A.1, Table V"},{"comment":"All results are reported as single point estimates, with no number of runs, error bars, or significance tests. This matters for claims of state-of-the-art or competitive performance where margins are small (e.g., WebQSP 73.8 vs. HGNet 70.6, or 73.8 vs. TERP 76.8). Please report the number of independent runs and variance, or at least clearly state that these are single-run results, so the reader can calibrate the comparisons.","section":"IV.A, Table IV"}],"minor_comments":[{"comment":"The sentence 'We also prove that this approach with our extensions can generalize well to other KGs without retraining' uses 'prove' for an empirical claim; no formal proof is provided, and the wording should be changed to 'demonstrate' or 'show.'","section":"II.B"},{"comment":"The notation D_A, G_A, D_B, G_B is introduced only after the table is referenced; please define it before the table and spell out that D_x is the data split and G_x is the ontology of dataset x.","section":"IV.A.1, Table V"},{"comment":"The training schedule ('first set a learning rate of 0.1 to train the learnable vectors and then set a learning rate of 5e-5 to train both the learnable vectors and the base model') is under-specified; state the number of epochs or steps for each phase, the batch size, and the criterion for switching phases to make the experiments reproducible.","section":"III.D"},{"comment":"The ablation compares prompt tuning (PT) with prompt tuning plus full fine-tuning (PT+FT), but it does not isolate the contribution of the four individual learnable vectors vQ, vG, vB, and vE; the claim that each vector captures 'aspects' of the input would be better supported by a per-vector ablation.","section":"IV.B, Table VII"},{"comment":"The limitations section is honest and relevant: relation directionality, SPARQL annotation style, and verbose Freebase naming conventions are real boundary conditions on the generalization claim. These limitations should be mentioned in the abstract or introduction so that 'generalize well' is not overpromised relative to the evidence presented.","section":"VI (Limitations)"}],"recommendation":"major_revision","confidential_remarks":"The central concern is whether the paper's cross-KG experiment actually demonstrates zero-shot transfer. The authors should be asked to clarify the Table VI protocol and, if targets were fine-tuned, to provide a genuine zero-shot cross-KG result or adjust the paper's claims. The ontology-prompt content issue (entities included in 'ontology') also needs clarification. No ethical concerns. The paper is within the scope of the journal."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a solid, well-written KGQA paper whose advertised headline—generalizing to unseen KGs without retraining—is not yet backed by the reported experiments. The architecture itself is worth taking seriously.\n\nWhat's genuinely new: the paper extends the structure/content two-stage idea (from Gu et al. and the authors' own prior work) by putting a verbalized ontology into both textual and learnable continuous prompts, adding a concept placeholder, and adding stage-specific decoding constraints (grammar, structure, subgraph). That combination is not in the cited literature. Within-dataset results in Table IV are competitive (79.1 F1 on LC-QuAD 1.0, 73.8 Hits@1 on WebQSP), and the ablation in Table VII shows the hybrid prompt matters. The limitations section is unusually candid.\n\nThe soft spot is the load-bearing generalization claim. The abstract says the system 'performs as well as SOTA without retraining' on CWQ, WebQSP, and LC-QuAD, but Table IV is per-dataset training—nothing cross-KG there. The zero-shot evidence is Table V, where the model gets 48.8 Hits@1 on CWQ and 34.0 F1 on SimpleDBpediaQA, far below SOTA, and there are no baseline numbers for these zero-shot conditions. Table VI, the only cross-KG experiment, never states whether the DBLP-QuAD or CoyPu training splits were used to fine-tune the model. 'Pre-trained variant' could easily mean 'initialized from LC-QuAD and then fine-tuned on the target,' which would make the +6.4/+3.1 gains supervised adaptation, not zero-shot transfer. As written, no experiment clearly supports 'without retraining' across KGs. This is fixable: state the protocol explicitly, add zero-shot baselines, and temper the abstract.\n\nMinor points: no error bars or significance tests on the within-dataset numbers; CoyPuKGQA has only 66 test questions, so the +3.1 margin is not very informative. The paper also does not compare against LLM-based KGQA methods common in recent work.\n\nBottom line: the method is coherent and the within-dataset evidence is real, but the central claim needs a clearer experimental story. I'd send it to review and ask for the missing protocol details and baseline comparisons. The authors are not sloppy; they just overclaim relative to what the tables show.","headline":"A credible two-stage KGQA architecture with honest within-dataset results, but the cross-KG 'without retraining' claim rests on an ambiguous table and needs clarification.","tokens_in":14679,"tokens_out":3214,"would_cite":true,"duration_ms":28611,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A two-stage SPARQL generator with ontology-guided hybrid prompts can answer questions over knowledge graphs it has never seen, without target training data.","keywords":["knowledge graph question answering","SPARQL query generation","ontology-guided prompting","hybrid prompt learning","cross-knowledge-graph generalization","large language models","constrained decoding"],"falsifier":"Take the released implementation, train on WebQSP only, then evaluate on the CoyPu KG test set twice: once with the true ontology in the prompt and once with the ontology replaced by a scrambled or unrelated ontology. If executed-query accuracy is essentially the same in both conditions, the claim that ontology guidance enables transfer is falsified; a large drop would support it.","tokens_in":13726,"feed_emoji":"🧩","tokens_out":9045,"duration_ms":78084,"temperature":0.7,"pith_summary":"This paper tries to establish that a knowledge graph question answering (KGQA) system can transfer to a new knowledge graph using only that graph's ontology and no target-specific training data. The proposed system, OntoSCPrompt, splits SPARQL generation into two stages: first it predicts a generic query skeleton with placeholders, then it fills in the entities, relations, and concepts specific to the target graph. The claimed mechanism is ontology-guided hybrid prompt learning, where the verbalized target ontology is prepended to the input together with a small set of learnable continuous vectors. On benchmarks such as CWQ, WebQSP, and LC-QuAD 1.0, the trained system performs on par with prior state of the art, and on unseen graphs such as DBLP-QuAD and CoyPu it reports usable F1 scores after pretraining on a different graph. The significance, if true, is that adapting to a new graph costs an ontology description rather than thousands of labeled question-query pairs.","feed_headline":"SPARQL skeleton first, content second: KGQA that generalizes","feed_subtitle":"Predict a SPARQL skeleton, then fill it from the target ontology to transfer without retraining","key_machinery":"The central object is the generic SPARQL query structure: a query skeleton with six placeholder tokens for entities, concepts, relations, variables, literals, and constraint conditions. It is what separates KG-independent semantics from KG-specific identifiers. Around it, ontology-guided hybrid prompt learning concatenates four learnable vectors with the embeddings of a textual prefix, the question, and the verbalized ontology, and tunes the whole with a negative log-likelihood objective. The skeleton makes cross-KG alignment explicit, while the ontology text and continuous vectors teach the model the target graph's naming conventions.","core_discovery":"OntoSCPrompt's central claim is that a two-stage decomposition of SPARQL generation into structure prediction and content population, steered by an ontology placed in hybrid prompts, makes KGQA transferable across heterogeneous knowledge graphs. In Stage-S the model outputs a query skeleton whose KG-specific slots are marked by six placeholders, and in Stage-C the model fills those slots using the target ontology and a retrieved subgraph. The paper reports that with only learnable prompt vectors and no target-KG fine-tuning, the model transfers within the same KG, such as a WebQSP-trained model reaching 48.8% Hits@1 on CWQ, and that fine-tuning on the target raises it to 70.4%. Across KGs, a model pretrained on LC-QuAD 1.0 improves F1 on DBLP-QuAD from 78.2 to 84.6 and on CoyPuKGQA from 80.2 to 83.3. The paper credits the constrained decoding strategies, together with the ontology-guided prompts, for keeping the generated SPARQL grammatical, structurally consistent, and executable.","pith_inferences":["Editorial inference: the strongest evidence for the core claim is the zero-shot within-KG transfer row, since the fine-tuned-on-target rows use target training data and should not be read as evidence of generalization.","Editorial inference: the method's ceiling depends on the ontology verbalization and the subgraph retriever; a direct test would degrade or scramble the ontology text and measure how much of the cross-KG gain disappears.","Editorial inference: because the structure stage is KG-agnostic, the same skeleton/content split could be applied to other structured query languages, such as SQL with schema prompts, whenever the target schema can be verbalized.","Editorial inference: very large ontologies may exceed the model's context window, so a natural extension is to retrieve a relevant ontology subset per question instead of providing the full verbalized ontology."],"forward_implications":["If the central claim holds, adapting a KGQA system to a new knowledge graph reduces to obtaining the target ontology, not collecting thousands of labeled question-SPARQL pairs.","The six-placeholder skeleton covers single-hop, multi-hop, aggregate, and constraint-bearing queries, so the claimed generalization is not restricted to simple lookups.","The constrained decoding strategies are load-bearing: grammar constraints, structure-guided pruning, and subgraph constraints each contribute to the reported accuracy on WebQSP.","Pretraining on a general-domain KG transfers positively to low-resource domain-specific KGs, with reported gains of +6.4 F1 on DBLP-QuAD and +3.1 on CoyPuKGQA.","Prompt tuning alone, without fine-tuning the base model, already reaches 70.3% F1 on LC-QuAD 1.0 and 62.1% Hits@1 on WebQSP, which supports the paper's claim that the hybrid prompts carry much of the KG-adaptation load."],"supporting_citations":[{"why":"Supplies the two-stage question-to-query decomposition that OntoSCPrompt extends and uses as its primary generalization baseline.","marker":"[18]"},{"why":"Supplies the structure-and-content prompt-learning formulation, including learnable continuous vectors and the negative log-likelihood training objective.","marker":"[19]"},{"why":"Supplies the original SPARQL structure/content preprocessing and placeholder scheme that this paper extends with a concept placeholder and complex clauses.","marker":"[22]"},{"why":"Supplies the ontology verbalization method that converts the target KG ontology into the text prepended to the prompts.","marker":"[23]"},{"why":"Supplies the subgraph retriever that provides question-relevant context and feeds the subgraph-constrained decoding in the content stage.","marker":"[34]"},{"why":"Supplies the long-context sequence-to-sequence Transformer used as the base model for both stages.","marker":"[33]"},{"why":"Supplies the DBLP-QuAD dataset and live SPARQL endpoint used to evaluate cross-KG generalization to an unseen scholarly knowledge graph.","marker":"[21]"}],"fun_headline_variants":["Split SPARQL into skeleton and fill: KGQA transfers across graphs","Ontology prompts let KGQA answer unseen graphs without retraining","Two-stage SPARQL: predict structure, fill from ontology for zero-shot KGQA","KGQA that learns a SPARQL skeleton, then fills slots from any KG's ontology","Decouple structure from content: ontology-guided prompts make KGQA portable"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that a short textual description of a knowledge graph's schema, plus subgraph context retrieved from that graph, is enough for a model trained on a different graph to write correct SPARQL queries against the new graph without ever seeing target training examples.","fun_headline_variants_meta":{"raw":{"variants":["Split SPARQL into skeleton and fill: KGQA transfers across graphs","Ontology prompts let KGQA answer unseen graphs without retraining","Two-stage SPARQL: predict structure, fill from ontology for zero-shot KGQA","KGQA that learns a SPARQL skeleton, then fills slots from any KG's ontology","Decouple structure from content: ontology-guided prompts make KGQA portable"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000672,"raw_usage":{"total_tokens":3115,"prompt_tokens":1051,"completion_tokens":2064,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":667,"completion_tokens_details":{"reasoning_tokens":1964}},"tokens_in":667,"tokens_out":2064,"duration_ms":14692,"temperature":1.0,"reasoning_tokens":1964,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T23:55:41.227455+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the released implementation, train on WebQSP only, then evaluate on the CoyPu KG test set twice: once with the true ontology in the prompt and once with the ontology replaced by a scrambled or unrelated ontology. If executed-query accuracy is essentially the same in both conditions, the claim that ontology guidance enables transfer is falsified; a large drop would support it.","supporting_citations":[{"cited_title":"A two-stage approach towards generalization in knowledge base question answering,","cited_arxiv_id":null,"evidence_quote":"Supplies the two-stage question-to-query decomposition that OntoSCPrompt extends and uses as its primary generalization baseline."},{"cited_title":"Few- shot text-to-sql translation using structure and content prompt learning,","cited_arxiv_id":null,"evidence_quote":"Supplies the structure-and-content prompt-learning formulation, including learnable continuous vectors and the negative log-likelihood training objective."},{"cited_title":"A structure and content prompt- based method for knowledge graph question answering over scholarly data,","cited_arxiv_id":null,"evidence_quote":"Supplies the original SPARQL structure/content preprocessing and placeholder scheme that this paper extends with a concept placeholder and complex clauses."},{"cited_title":"Text2kgbench: A benchmark for ontology-driven knowledge graph generation from text,","cited_arxiv_id":null,"evidence_quote":"Supplies the ontology verbalization method that converts the target KG ontology into the text prepended to the prompts."},{"cited_title":"Subgraph retrieval enhanced model for multi-hop knowledge base question answering,","cited_arxiv_id":null,"evidence_quote":"Supplies the subgraph retriever that provides question-relevant context and feeds the subgraph-constrained decoding in the content stage."},{"cited_title":"LongT5: Efficient text-to-text transformer for long sequences,","cited_arxiv_id":null,"evidence_quote":"Supplies the long-context sequence-to-sequence Transformer used as the base model for both stages."},{"cited_title":"Dblp-quad: A question answering dataset over the DBLP scholarly knowledge graph,","cited_arxiv_id":null,"evidence_quote":"Supplies the DBLP-QuAD dataset and live SPARQL endpoint used to evaluate cross-KG generalization to an unseen scholarly knowledge graph."}],"review_version":1}