{"id":"ec289b7f-e826-4c80-9a8a-194343d61081","arxiv_id":"2412.13688","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Ontology competency questions can be classified into five types, Scoping, Validating, Foundational, Relationship, and Metaproperty, each with distinct purposes and components.","lead":"This paper proposes a taxonomy of five types of competency questions used in building ontologies, each serving a different purpose from scoping to validating to metaproperty analysis. It also introduces a public repository of 438 annotated example questions to support ontology engineers.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The five CQ types are not shown to be mutually exclusive or reliably assignable: SCQ and VCQ definitions overlap, so type membership depends on use context rather than on question constituents.","rationale":"I read the paper as a first conceptual taxonomy, and the repository plus user story are useful contributions; the authors are also transparent about the dearth of examples for the newer types. The main risk is not internal inconsistency in the narrative, but that the central claim — that the five types are distinguishable by their constituent elements — is not operationalised. Because the same surface CQ can be used for scoping or for validation, the type is a property of the use, not of the question itself. The definitions encode this by introducing purpose predicates whose criteria are not specified, making the taxonomy hard to test or apply consistently. The proposed inter-annotator study would settle whether trained ontology engineers can reliably assign types from the definitions alone. This is the same load-bearing assumption the reader identified, and it supports keeping the verdict conditional rather than accepting the central claim as established. No change to the reader's verdict is needed.","tokens_in":12350,"tokens_out":4053,"duration_ms":40595,"concrete_test":"Select a stratified random sample of 100 CQs from ROCQS, remove their assigned types and all contextual information, and have at least three ontology engineers independently classify each CQ into one or more of the five types using only Definitions 1-6. Compute Fleiss' kappa for multi-label agreement. If mean kappa is below 0.6, or if more than 10% of items are assigned multiple types, then the claim that types are discernible by constituent elements and mutually exclusive is not supported. If kappa is high and multi-type assignments are rare, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that CQs can be discerned and characterised by type, with each type having identifiable distinct constituent elements. The paper's definitions (Defs 2-6) are only necessary conditions: each has the form ∀x(T(x) → ∃...), with no conditions excluding other types. Def 2 (SCQ) requires mention of a DomainEntity and usage with an Ontology and SubjectDomain; Def 3 (VCQ) requires mention of a DomainEntity and validate with an Ontology. A single sentence such as \"Which animals are endangered?\" (used as the SCQ example in §3.2) also satisfies Def 3 if asked during verification; the paper itself states that most SCQs are reused for validation. The distinction between SCQ and VCQ therefore rests on unformalised purpose predicates (usage vs validate), not on identifiable constituent elements. This is acknowledged in the text (\"subtle differences ... with respect to intent or purpose\"), but it undermines the stated claim that each type has identifiable distinct constituent elements. Similar overlap is plausible for other pairs, and no disjointness axioms are given. The formalisation also contains a free variable w in the drRCQ formula (§3.2), indicating the logical characterisation is not yet a reliable basis for classification. Without an operational coding procedure or inter-annotator study, the taxonomy remains a plausible conceptual proposal rather than an empirically supported model. This directly affects the paper's primary contribution, not merely its presentation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a model, QuO, for competency questions (CQs) in ontology engineering, claiming that CQs can be discerned and characterised into five principal types: Scoping (SCQ), Validating (VCQ), Foundational (FCQ), Relationship (RCQ), and Metaproperty (MpCQ). Each type is given an informal definition, an EER-style diagram, and a first-order logic formalisation. The paper illustrates the model with a coffee-ontology user story, maps the types onto stages of the NeOn methodology, discusses faulty CQs, and introduces ROCQS, a FAIR repository of 438 annotated CQs. The central claim is that the five types have distinct purposes and identifiable distinct constituent elements, enabling better authoring, validation, and deployment of CQs in ontology development.","tokens_in":12689,"tokens_out":4357,"duration_ms":38770,"significance":"The paper addresses a real gap: although CQs are widely used in ontology engineering, there is little theoretical guidance on their types and constituents. The proposed taxonomy is a useful conceptual contribution if the types can be reliably distinguished, and the accompanying ROCQS repository is a concrete, reusable artifact that can support further empirical work. The paper is also commendable for making its definitions explicit, for attempting formal characterisations, and for connecting the types to methodology stages in NeOn. However, the central claim is currently supported mainly by author-generated examples and a single illustrative user story; there is no inter-annotator study or operational coding procedure. The definitions, as written, do not yet establish that the five types are mutually exclusive or reliably assignable. The significance therefore rests on the plausibility of the taxonomy rather than on demonstrated empirical support, and the formal apparatus contains technical gaps that need to be addressed before it can serve as a reliable basis for classification.","major_comments":[{"comment":"The definitions of SCQ and VCQ do not distinguish the two types. Definition 2 requires mention of a domain entity and use with an ontology and subject domain; Definition 3 requires mention of a domain entity and validation with an ontology. The only formal difference is the unformalised primitive usage versus validate. The paper itself notes that most SCQs are reused for validation, and the example 'Which animals are endangered?' satisfies both definitions. Thus the claimed 'identifiable distinct constituent elements' are not captured by the formalisations. If the types are distinguished by intent or purpose, that predicate must be made explicit, or the taxonomy should be presented as a purpose-based idealisation rather than a constituent-based classification. This is load-bearing because the central claim is that the types are discernible from their constituents.","section":"§3.2, Definitions 2 and 3"},{"comment":"The first-order logic formalisation of drRCQ contains an unbound variable w: the formula states Ontology(w) and containsVocab(w,y) and containsVocab(w,z) without quantifying w. In addition, the expressiveness constraint in Definition 3, L_v ⊆ L_o, is asserted as sufficient for VCQ answerability without proof; answerability also depends on content coverage and on the query language used. These issues matter because the formalisations are presented as evidence that the types are distinct and can be operationalised.","section":"§3.2, drRCQ formalisation and Definition 3"},{"comment":"The evaluation does not support the claim that the taxonomy can be reliably discerned. Section 4.1 presents a single user story with examples authored for the purpose, and Section 5 acknowledges that the authors 'needed to design templates and generate CQs for other categories.' There is no inter-annotator study, no coding procedure, and no independent corpus annotation beyond the authors' own repository. The paper should either provide an operational annotation protocol with agreement measures or explicitly reposition the taxonomy as a hypothesis whose empirical validation is future work. Without this, the repository's type labels cannot be taken as evidence of distinct types.","section":"§4.1 and §5"},{"comment":"The paper asserts that 'the ontological CQs—aRCQ, efRCQ, rpRCQ, MpCQ—relate to an entity and the answer to the CQ ought to necessarily hold across all ontologies.' This strong claim is not established by the definitions. For example, a rpRCQ is defined with respect to a particular ontology o and requires that the relational property be expressible in the representation language of o, which makes the answer ontology-relative. Please either justify the universality claim or qualify it, since it is used to divide ontological from domain CQs.","section":"§3.2, closing comparison of CQ scope"}],"minor_comments":[{"comment":"The formalisation contains a typo: 'SQC' should be 'SCQ'.","section":"§3.2, Definition 2"},{"comment":"The bullet list uses 'FQCs' where 'FCQs' is intended.","section":"§4.1"},{"comment":"The methodology name is written inconsistently as both 'NeON' and 'NeOn'; please standardise.","section":"Throughout"},{"comment":"The repository is referenced by a URL only; a DOI or other stable identifier would strengthen the FAIR claim.","section":"§4.3"},{"comment":"The distinction between VCQs and the listed 'constraint-checking queries' is unclear, since the example 'Are there animals that are both a carnivore and a herbivore?' could also be read as a VCQ; clarifying the criterion would help readers apply the taxonomy.","section":"§4.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for a knowledge-engineering venue and the repository is a useful contribution. The main risk is that the taxonomy's empirical support is thin; I would be willing to consider a revised version that adds an annotation study or clearly reframes the taxonomy as a hypothesis. I also note that the self-authored examples are acknowledged in the text but should be surfaced more prominently so that readers do not mistake them for independent evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a look if you work in ontology engineering. The paper gives a genuinely new five-type taxonomy for competency questions—Scoping, Validating, Foundational, Relationship, Metaproperty—with definitions, EER diagrams, and a public repository of 438 annotated CQs. That is a real contribution. The repository alone is a useful resource, and the mapping of each type onto NeOn methodology stages helps practitioners see where these questions fit.\n\nThe paper is honestly written. It credits prior work (Uschold and Gruninger, Wisniewski et al., recent LLM CQ papers), and it admits that SCQs and VCQs differ mainly in intent and that the new types had to be generated because few examples existed. That candor is appreciated.\n\nThe soft spots sit in the strength of the central claim. The formal definitions are only necessary conditions: nothing prevents a question from being both an SCQ and a VCQ, and the paper's own example 'Which animals are endangered?' fits both depending on when it is asked. So the types are not shown to be distinguished by their constituent elements; intent does a lot of work. There is also an unbound variable in the drRCQ formula (the w), so the logic is not yet a classifier. The evaluation is a single hypothetical user story, and many of the new examples were invented by the authors to fit the definitions, which gives the distinctness claim a bit of a circular feel. No inter-annotator study or coding procedure backs the taxonomy.\n\nNone of this is fatal. The taxonomy is plausible, the repository gives it grounding, and the paper is specifically useful to ontology engineers who want clearer vocabulary for CQ authoring and validation. The main fix is to either soften the distinctness claim or add an empirical coding study. I'd be happy to see this go to peer review—it deserves referee time. I'd probably cite it when discussing CQ types, and I'd bring it up in a reading group focused on ontology methods.","headline":"A genuinely new five-type taxonomy for competency questions and a public repository, but the boundaries between types rest more on intent than on question text, so the distinctness claim needs qualification.","tokens_in":13150,"tokens_out":3367,"would_cite":true,"duration_ms":29502,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper introduces a first model that classifies competency questions for ontologies into five types—scoping, validating, foundational, relationship, and metaproperty—each with its own purpose and required components.","keywords":["competency questions","ontology engineering","ontology development","scoping","validation","foundational ontology alignment","metaproperties","ROCQS"],"falsifier":"Look at the 438 questions in ROCQS and have two or more ontology engineers, working only from the paper's definitions, assign each question to exactly one type; if agreement is low or many questions are assigned to multiple types, the claimed distinctness is not present in the data.","tokens_in":12154,"feed_emoji":"❓","tokens_out":8049,"duration_ms":68425,"temperature":0.7,"pith_summary":"Competency questions (CQs) are widely used to scope and validate ontologies, but engineers get little guidance on what makes a good one. This paper tries to establish that CQs are not all alike: they serve at least five distinguishable purposes, and each purpose can be recognised by characteristic components in the question. It proposes the Questions for Ontologies (QuO) model, which defines Scoping, Validating, Foundational, Relationship, and Metaproperty CQs, each with a formal characterisation and a diagrammatic specification. The authors support the model with a coffee-ontology user story, a mapping onto stages of a standard ontology-engineering methodology, and a FAIR repository of 438 annotated CQs (ROCQS). If the model is right, ontology engineers gain a vocabulary for saying what a CQ is for and what it must contain, which should reduce ambiguity and improve ontology quality.","feed_headline":"First taxonomy gives ontology competency questions five distinct types","feed_subtitle":"Scoping, validating, foundational, relationship, and metaproperty questions each have distinct parts to guide ontology design.","key_machinery":"The carrying object is the QuO model itself: a taxonomy of five CQ types anchored by formal definitions (Definitions 1 to 6) and entity-relationship diagram snippets that specify the mandatory constituents of each type. The key components are the primitives DomainEntity, SubjectDomain, Ontology, foundational-ontology elements, Relationship, Metaproperty, and Metametaproperty, plus the closed sets of relational properties and metaproperties; the definitions use these to give each type a distinct purpose and a distinguishing shape. The model is put to work by annotating a standard development methodology with the CQ types relevant at each stage and by populating ROCQS, a FAIR repository of 438 CQs with examples and templates for each type.","core_discovery":"The paper's central claim is that the apparent jumble of competency questions in ontology development resolves into a first model with five principal types. Scoping CQs delimit the subject domain and must mention a domain entity and a subject domain; Validating CQs test an ontology's content and must be formalisable and answerable within the ontology's language; Foundational CQs interrogate a domain entity against the vocabulary of a foundational ontology to support alignment; Relationship CQs probe a relationship's arity, elementariness, participants, or relational properties; Metaproperty CQs classify an entity according to metametaproperties such as rigidity, identity, unity, and dependence. Each type is given a formal definition with distinct constituent elements, and the paper argues this accounts for the different uses of CQs documented in the literature. It further demonstrates where each type fits in ontology-development tasks and supplies an annotated repository, ROCQS, containing 438 CQs as evidence and as a resource.","pith_inferences":["Because the formal definitions give necessary conditions rather than a decision procedure, the five types are likely to overlap in practice: one and the same question can serve scoping and validation, so the model may be most useful as a multi-label characterisation rather than a partition.","The same component-based method could be carried over to other requirements-engineering settings: classify any requirement question by purpose and mandatory constituents, then use the type to guide repair; this would test whether the underlying idea generalises beyond ontologies.","A low-cost experiment the paper does not run is to give the definitions to independent ontology engineers and measure whether they assign repository questions to the same types; a positive result would turn the model into an operational coding scheme."],"forward_implications":["Developers can choose a CQ type by task: scoping during requirements, validation during implementation, foundational alignment when connecting to upper-level ontologies, and metaproperty or relationship questions during formalisation.","Validating CQs come with an expressiveness constraint: a question only qualifies if the ontology language can express and answer it, which gives a concrete criterion for dropping unusable CQs.","Foundational CQs make alignment errors diagnosable: a question is faulty when it asks for distinctions the target foundational ontology does not make.","Relationship and metaproperty CQs are answered by the modeller through ontological analysis rather than by querying the ontology, changing what tool support for them should look like.","The ROCQS repository provides a shared, annotated pool of 438 CQs whose type labels can be used to evaluate automated CQ generation and authoring tools."],"supporting_citations":[{"why":"Supplies the original characterisation of competency questions as specifying ontology requirements and the design search space, which the paper's five-type model extends.","marker":"[25]"},{"why":"Provides the analysed dataset of 234 informal competency questions and their SPARQL-OWL translations, which motivates and feeds the new repository.","marker":"[28]"},{"why":"Is the development methodology whose stages the paper annotates with the type of CQ relevant at each step.","marker":"[23]"},{"why":"Establishes test-driven development of ontologies, the practice that grounds the validating CQ type.","marker":"[15]"},{"why":"Documents, through a survey, that CQs are used mainly for scoping and evaluation and that missing guidelines are a known difficulty, motivating the model.","marker":"[20]"},{"why":"Supplies foundational-ontology alignment questions used as examples of Foundational CQs.","marker":"[3]"},{"why":"Provides a controlled language for authoring CQs, which the paper notes does not cover foundational or metaproperty question types.","marker":"[16]"},{"why":"Defines the formal ontology of properties whose set of metaproperties the Metaproperty CQ definitions rely on.","marker":"[10]"},{"why":"Introduces the metaproperty toolkit of rigidity, identity, unity, and dependence that Metaproperty CQs interrogate.","marker":"[11]"}],"fun_headline_variants":["Five types classify competency questions for ontology design","Ontology competency questions get a five-type framework","Competency questions: five distinct types to guide ontologies","New model sorts ontology competency questions into five types","A first model for competency questions: five types defined"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The taxonomy assumes that a competency question's purpose can be read off from its wording and context, and that those purposes sort cleanly into five categories that different ontology engineers would agree on.","fun_headline_variants_meta":{"raw":{"variants":["Five types classify competency questions for ontology design","Ontology competency questions get a five-type framework","Competency questions: five distinct types to guide ontologies","New model sorts ontology competency questions into five types","A first model for competency questions: five types defined"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000163,"raw_usage":{"total_tokens":1268,"prompt_tokens":992,"completion_tokens":276,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":608,"completion_tokens_details":{"reasoning_tokens":203}},"tokens_in":608,"tokens_out":276,"duration_ms":2894,"temperature":1.0,"reasoning_tokens":203,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T12:53:11.614586+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Look at the 438 questions in ROCQS and have two or more ontology engineers, working only from the paper's definitions, assign each question to exactly one type; if agreement is low or many questions are assigned to multiple types, the claimed distinctness is not present in the data.","supporting_citations":[{"cited_title":"The Knowl- edge Engineering Review 11(2), 93–136 (1996)","cited_arxiv_id":null,"evidence_quote":"Supplies the original characterisation of competency questions as specifying ontology requirements and the design search space, which the paper's five-type model extends."},{"cited_title":"Journal of Web Semantics 59, 100534 (2019)","cited_arxiv_id":null,"evidence_quote":"Provides the analysed dataset of 234 informal competency questions and their SPARQL-OWL translations, which motivates and feeds the new repository."},{"cited_title":"NeOn Deliverable D5.4.1, NeOn Project (2008)","cited_arxiv_id":null,"evidence_quote":"Is the development methodology whose stages the paper annotates with the type of CQ relevant at each step."},{"cited_title":"In: Sack, H., et al","cited_arxiv_id":null,"evidence_quote":"Establishes test-driven development of ontologies, the practice that grounds the validating CQ type."},{"cited_title":"In: Proc","cited_arxiv_id":null,"evidence_quote":"Documents, through a survey, that CQs are used mainly for scoping and evaluation and that missing guidelines are a known difficulty, motivating the model."},{"cited_title":"In: Proc","cited_arxiv_id":null,"evidence_quote":"Supplies foundational-ontology alignment questions used as examples of Foundational CQs."},{"cited_title":"In: Proc","cited_arxiv_id":null,"evidence_quote":"Provides a controlled language for authoring CQs, which the paper notes does not cover foundational or metaproperty question types."},{"cited_title":"In: Dieng, R., Corby, O","cited_arxiv_id":null,"evidence_quote":"Defines the formal ontology of properties whose set of metaproperties the Metaproperty CQ definitions rely on."},{"cited_title":"In: Horn, W","cited_arxiv_id":null,"evidence_quote":"Introduces the metaproperty toolkit of rigidity, identity, unity, and dependence that Metaproperty CQs interrogate."}],"review_version":1}