{"id":"0af3774e-56dc-4d8d-ac3f-407c153f8aa2","arxiv_id":"2607.17963","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A retrieval-augmented LLM framework extends existing ontologies from competency questions, generating fragments that pass structural and functional checks and are judged usable as drafting aids by ontology engineers.","lead":"OntoExtend is a framework that uses large language models to extend existing ontologies: it retrieves the most relevant parts of an ontology for a new competency question, then prompts an LLM to generate the missing classes and properties. In tests on 39 questions across an EU circular-economy ontology and a Bosch manufacturing ontology, the generated fragments passed structural and functional checks and ontology engineers rated them as needing minor to moderate revision.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"CQ verification is post-hoc: authors write SPARQL after seeing the generated fragment, so the 100% verification rate and <2% superfluous count do not establish that the intended model was recovered.","rationale":"After reading the paper, the central claim is that OntoExtend offers practically useful automation because generated fragments are structurally clean, pass functional tests, and have <2% superfluous elements. The human-survey results provide genuine evidence of usefulness, especially for the industry ontology, but the paper's two quantitative functional claims are both anchored in a SPARQL-based test that is authored post hoc by the same team. This is the least secure load-bearing condition: if the verification query can be written after the fact, '100% CQ verification' no longer means the intended domain was modelled, and the superfluous count inherits the same bias. The authors do disclose data-leakage risk and the dependence on CQ quality, and they acknowledge that EU-project CQs were open-ended; those admissions soften but do not repair the circularity. The availability footnote and §7 limitations also conflict, which matters because an independent check of the CQs is impossible as described. I am not recommending rejection: the framework and evaluation design are otherwise clear, and the industry use case has independent expert ratings. But the conditional acceptance should explicitly require pre-registered gold queries or blind query authoring, and a reconciled data-availability statement. This matches the reader's weakest assumption, so no verdict change is needed beyond the existing CONDITIONAL.","tokens_in":13235,"tokens_out":6488,"duration_ms":57081,"concrete_test":"Pre-register, for all 39 CQs, gold-standard SPARQL queries and expected answer sets derived from the original ontologies before any elements are removed; freeze them before generation. After integrating each generated fragment, execute the frozen queries unchanged. If fewer than 100% retrieve exactly the expected answers, the reported CQ-verification rate is inflated by post-hoc query writing. Recompute the superfluous-element count against the full set of removed classes/properties rather than against the verification query; if it exceeds 2%, the 'fewer than 2%' claim also fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative support for 'practically useful automation' rests on CQ verification and the superfluous-element count. Section 4.1 defines CQ verification as 'attempting to formulate a SPARQL query that retrieves an answer for that CQ'; a CQ is 'considered not modelled' only if no such query can be written. This is a post-hoc test: the same team that created the CQs and the expected answers also writes the verification query after the LLM has generated the fragment. There is no pre-registered gold query, and the query can be adapted to whatever the fragment happens to contain. A semantically wrong fragment—e.g., an inverse property or mis-scoped class—can still be queried successfully. Section 5.2 says two ontology engineers verify and cross-check, but these are the authors, not blind evaluators. The superfluous-element definition in Section 4.1 is explicitly relative to 'the verification SPARQL used in CQ verification,' so the second quantitative pillar is not independent of the first. The human survey is more convincing (industry ratings 4.5–4.9, EU 2.9–3.7), but it supports 'minor to moderate revision,' not the paper's claim that all CQs are 'accurately modelled.' The §7 limitation about data leakage is acknowledged, but the availability statement in fn. 1 (CQs available) conflicts with §7 (CQs not publicly available; industry ontology not disseminated), further blocking independent replication. This is a load-bearing concern because if the post-hoc-query objection lands, the headline quantitative results lose their force; the framework may still be useful, but the current evidence is weaker than claimed.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces OntoExtend, a retrieval-augmented generation (RAG) framework for extending existing ontologies with LLMs, driven by competency questions (CQs). The pipeline consists of an Ontology Retriever that indexes and retrieves relevant elements from input ontologies, an Ontology Extender that prompts an LLM to generate Turtle fragments, and an Ontology Integrator that merges fragments back into the source ontologies. The framework is evaluated on 39 CQs from two real-world settings: four modules of the public EU-project ontology Onto-DESIDE and an industrial Bosch ontology. The reported results indicate few structural pitfalls, 100% CQ verification, fewer than 2% superfluous elements, and engineer ratings of 4.5–4.9 for the industry setting and 2.9–3.7 for the EU setting. The paper claims OntoExtend provides a practically useful drafting-assistant level of automation for real-world ontology extension.","tokens_in":13623,"tokens_out":2660,"duration_ms":24621,"significance":"If the central claims hold, OntoExtend would be a meaningful step toward scalable, requirement-driven ontology extension with LLMs, particularly in its use of retrieval to ground generation in existing ontology elements. The paper has concrete strengths: it implements and releases a working framework, applies structural checks (Turtle syntax, OOPS!, Pellet consistency), and reports a human evaluation with inter-annotator agreement. These are valuable beyond the specific results. However, the quantitative support for 'practically useful automation' rests on two metrics—CQ verification and superfluous-element rate—that are defined circularly with respect to the authors' own post-hoc SPARQL queries, and the evaluation lacks a within-experiment baseline and uses CQs reverse-engineered from the target ontologies. The significance is therefore conditional: the framework is plausible and the industrial survey is encouraging, but the central evidence is not yet independently probative.","major_comments":[{"comment":"CQ verification is post-hoc. The definition states that a CQ is considered modelled if the authors can 'attempt to formulate a SPARQL query that retrieves an answer for that CQ' after seeing the generated fragment. The same team writes the CQs, the expected answers, and the verification queries, and the queries are not pre-registered. A semantically wrong fragment—e.g., one using an inverse property, a mis-scoped class, or an overly broad domain—can still be queried successfully. Therefore the reported 100% verification rate in Table 4 does not establish that the intended model was recovered. This is load-bearing because the abstract and Section 7 rely on this result.","section":"Section 4.1, Section 5.2, Section 6.2"},{"comment":"The superfluous-element metric is not independent of CQ verification. By definition, an element is superfluous if it is not mentioned in the verification SPARQL and is not a subclass/subproperty of a query component. Since the verification SPARQL is written after the fragment is generated, the <2% superfluous rate is partly an artifact of the authors' ability to write queries that fit the generated fragment. The comparison to the 'around 30%' in [16] is also not a controlled baseline: it compares different ontologies, different prompts, and different tasks. A within-experiment baseline is needed to support the claim that retrieval reduces superfluous elements.","section":"Section 4.1, Section 6.2"},{"comment":"The benchmark is constructed by removing classes and their related properties from the target ontologies and then asking what these removed elements were 'intended to represent or accomplish within the original ontology.' This assumes that removal-based CQs are representative of real extension requirements, and it makes the expected answers knowable to the authors by construction. Because the same group formulated the CQs, the answers, and the verification queries, the risk of circularity is high. Independent gold CQs, blind evaluation, or at least a preregistered query/answer set would be needed to treat the functional evaluation as evidence of correct modelling.","section":"Section 4, Dataset Creation"},{"comment":"The paper's central claim overstates the survey results. Section 6.3 states that 'the evaluators with high agreement concluded that the generated fragments require moderate revision before they can be safely integrated,' and Table 5 shows EU-project completeness means of 2.94–3.11, which on the paper's own scale corresponds to 'moderate changes' (11–25% revision), not 'minor' or 'light.' Yet Section 7 claims OntoExtend 'generates a complete extension module that users typically only need to review and lightly edit,' and the abstract similarly says 'minor to moderate revision.' The EU-project results support only a drafting-assistant claim with nontrivial post-editing effort in open-CQ settings; the 'lightly edit' formulation is not supported by the reported data.","section":"Section 6.3, Section 7, Section 8"},{"comment":"There is a direct contradiction in the availability statement. Footnote 1 says 'The code and the experiment data (ontologies and CQs), along with examples ... are available at https://github.com/...' while Section 7 states 'for the EU-Project ontology, the ontologies have been publicly released, but the material concerning CQs and their coverage is not publicly available' and that the industry ontology 'was not disseminated in public code or data repositories.' This inconsistency blocks independent replication and must be resolved by making the actual artifacts available or clearly specifying what is and is not released.","section":"Footnote 1 vs Section 7, Limitations"}],"minor_comments":[{"comment":"Typo: 'For each entity. It constructs an OntologyElement record' should be 'For each entity, it constructs...'.","section":"Section 3.1"},{"comment":"'cross cross-checking' is a duplication; should be 'cross-checking'.","section":"Section 4, Parameter tuning"},{"comment":"The OOPS! column header 'P2, P4 & P8' is not defined in the caption; it should state that these are minor pitfall types and clarify whether the counts are per-fragment or per-use-case.","section":"Table 4"},{"comment":"The 'Element to be embedded' example in Figure 2 is described in text but the figure itself is not referenced in a way that clarifies the pipe-delimited serialization; consider showing a full example inline.","section":"Section 3.1"},{"comment":"The survey rating scale for Completeness is ordinal but means are reported; no standard deviation or per-evaluator distribution is given, making it hard to assess the spread behind the Fleiss P_o values.","section":"Section 4.1"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely and practical problem, and the framework design is sensible. However, the evaluation methodology has a circularity problem: the CQ verification and superfluous-element metrics are defined relative to post-hoc queries written by the same team, and the benchmark is derived from the target ontologies themselves. These issues are load-bearing for the central 'practically useful' claim and require substantive changes—not just text edits. If the authors can add a controlled baseline, pre-registered or blind evaluation, or otherwise decouple the metrics from the generation, the paper could become acceptable. The availability contradiction between footnote 1 and Section 7 is also a serious reproducibility concern that should be fixed before any acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: OntoExtend is a real contribution—a retrieval-extender-integrator pipeline for extending ontologies from competency questions, with explicit reuse of existing elements. It's clearly described, and the human survey from industry is genuinely encouraging. But the headline numbers (100% CQ-verification, <2% superfluous) are built on a post-hoc verification procedure: the authors write the SPARQL after seeing the generated fragment. That doesn't tell you whether the intended model was recovered.\n\nWhat's new: the combination of RAG over the target ontology, CQ-driven prompting, and integration with re-indexing is not in the related work they cite, which mostly does taxonomy enrichment or from-scratch generation. The engineering is thoughtful—prompt templates, configurable profiles, two-stage validation. And the evaluation is more thorough than most LLM-ontology papers: OOPS!, Pellet, syntax, human raters.\n\nWhere it's soft: First, the functional test. Section 4.1 defines CQ verification as 'attempting to formulate a SPARQL query'—so a fragment is 'correct' as long as the same team that wrote the CQ can find any query that retrieves an answer. A wrong fragment can still be queried. Second, the superfluous-element count is relative to that same verification query, so the two quantitative pillars are not independent. Third, there's no within-experiment baseline (e.g., no retrieval, or full ontology in context). Fourth, the benchmark CQs are reverse-engineered by removing elements from the target ontologies, which may not reflect real extension requirements. Fifth, the availability statement conflicts: footnote says CQs are available on GitHub; §7 says they're not publicly available. That blocks replication.\n\nThe human survey is the most believable evidence, and it supports 'drafting assistant'—industry ratings around 4.5-4.9, EU around 3. That is not the same as 'accurately model all CQs.' The authors acknowledge leakage risk and CQ sensitivity honestly. So the framework is probably useful; the claims about automation are stretched.\n\nBottom line: this deserves a serious referee, not a desk reject. A good revision would add a no-retrieval baseline, pre-specify gold SPARQL queries (or use independent, blinded evaluators), fix the availability inconsistency, and soften the conclusions to match the survey scores.","headline":"A solid engineering framework for CQ-driven ontology extension, but the 100% verification and <2% superfluous numbers are undercut by post-hoc evaluation; the paper deserves peer review with major revisions.","tokens_in":14109,"tokens_out":4443,"would_cite":true,"duration_ms":35849,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"OntoExtend uses retrieval-grounded LLM generation to extend existing ontologies, passing all 39 competency questions with under 2% superfluous elements and few structural defects.","keywords":["ontology extension","large language models","retrieval-augmented generation","competency questions","ontology evaluation","SPARQL verification","ontology reuse","SHACL"],"falsifier":"Have a team uninvolved in the design of a test ontology write the CQs and, independently, have annotators check whether the generated fragments correctly model the intended meaning without relying on SPARQL-answerability. If a substantial fraction of fragments pass the paper's verification but are judged semantically wrong by independent annotators, the central claim of practical automation is undermined.","tokens_in":13155,"feed_emoji":"🧩","tokens_out":4007,"duration_ms":32482,"temperature":0.7,"pith_summary":"OntoExtend is a framework for extending existing ontologies with the help of large language models (LLMs), driven by new requirement questions called competency questions (CQs). It retrieves the most relevant classes, properties, and axioms from the ontology, feeds them to an LLM as grounded context, and generates a small fragment of new axioms that extend the ontology to answer the CQ. The paper evaluates this on 39 CQs across two real ontologies, finding that fragments are syntactically valid, introduce almost no modelling pitfalls, contain fewer than 2% superfluous elements, and all CQs pass a SPARQL-based functional test. Ontology engineers rated the outputs as needing only minor to moderate revision, supporting the claim that OntoExtend can act as a practical drafting assistant for ontology maintenance.","feed_headline":"Grounded LLMs draft ontology extensions that pass all 39 checks","feed_subtitle":"Retrieval-grounding in the existing ontology yields clean fragments that engineers rate as needing only minor to moderate edits.","key_machinery":"The central mechanism is retrieval-augmented generation (RAG) over the ontology itself: each class, property, and axiom is serialized as a pipe-delimited text element, embedded with a sentence model, and stored in a vector index; a new competency question is embedded and the top-k most similar elements are retrieved and injected into the LLM prompt as Turtle snippets. This grounding step is what keeps the generated axioms consistent with the existing ontology's terminology and modelling patterns, and is what the authors credit for the low structural defect rate and the near-absence of superfluous elements.","core_discovery":"The authors claim that ontology extension — enriching an existing ontology to satisfy new requirements — can be largely automated without sacrificing quality, if the LLM's generation is grounded in the specific ontology elements that are relevant to each requirement. Their pipeline retrieves top-k relevant ontology entities via dense vector embeddings of element descriptions, injects the corresponding Turtle snippets into the prompt, and generates a self-contained axiom fragment. On two real-world cases, the fragments were structurally clean, verified against all 39 competency questions, added under 2% superfluous elements, and received positive human ratings, leading the authors to conclude","pith_inferences":["Because the same team crafted the removed classes, the CQs, and the verification SPARQL queries, the 100% CQ-verification rate may overestimate semantic correctness; an independent evaluation with third-party CQs would test this.","The retrieval step is text-based (serialized labels, comments, and axioms); a graph-structural retriever that exploits relational neighbours might retrieve different elements for CQs that require deep inference chains.","Enabling the optional re-indexing of generated fragments (disabled in this evaluation) could improve cross-fragment consistency, but would also make errors propagate; this is a testable design choice.","The authors' interpretation of LLM behaviour as a proxy for CQ quality suggests a practical diagnostic: poorly performing extensions flag under-specified requirements, which could be used to prompt better requirement elicitation."],"forward_implications":["Mature ontologies can be extended incrementally as new requirements arise, with LLM-generated drafts that engineers review instead of authoring from scratch.","The retrieval step makes the approach scalable to ontologies far larger than an LLM's context window, since only a compact relevant subset is sent to the model.","The framework is portable across domains and modelling profiles via configurable prompt templates (e.g., OWL restrictions vs. SHACL shapes), demonstrated on two different real-world settings.","All generated fragments passed the functional CQ-verification test and added fewer than 2% superfluous elements, a large improvement over reported rates in from-scratch LLM ontology generation.","LLM performance on extension appears sensitive to CQ specificity, suggesting that quality of requirements is a key downstream factor."],"fun_headline_variants":["LLM ontology extension passes all 39 checks with RAG grounding","RAG-grounded LLM drafts clean ontology fragments for real use","Requirement-driven LLM extension: 39/39 checks passed","Grounded LLM ontology drafts rated minor to moderate edits","Ontology extension via RAG-grounded LLM passes all competency checks"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The evaluation assumes that competency questions formed by deleting classes from an ontology and asking what those classes were meant to represent faithfully capture real-world extension needs, and that a fragment is correct if a SPARQL query can be written to retrieve an answer for the CQ.","fun_headline_variants_meta":{"raw":{"variants":["LLM ontology extension passes all 39 checks with RAG grounding","RAG-grounded LLM drafts clean ontology fragments for real use","Requirement-driven LLM extension: 39/39 checks passed","Grounded LLM ontology drafts rated minor to moderate edits","Ontology extension via RAG-grounded LLM passes all competency checks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000686,"raw_usage":{"total_tokens":2929,"prompt_tokens":710,"completion_tokens":2219,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":454,"completion_tokens_details":{"reasoning_tokens":2129}},"tokens_in":454,"tokens_out":2219,"duration_ms":13106,"temperature":1.0,"reasoning_tokens":2129,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T16:29:30.935065+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have a team uninvolved in the design of a test ontology write the CQs and, independently, have annotators check whether the generated fragments correctly model the intended meaning without relying on SPARQL-answerability. If a substantial fraction of fragments pass the paper's verification but are judged semantically wrong by independent annotators, the central claim of practical automation is undermined.","supporting_citations":[],"review_version":1}