REVIEW 4 major objections 5 minor 28 references
Discerning and Characterising Types of Competency Questions for Ontologies
T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read This paper introduces a first model that classifies competency questions for ontologies into five types—scoping, validating, foundational, relationship, and metaproperty—each with its own purpose and required components.
desk verdict A genuinely new five-type taxonomy for competency questions and a public repository, but the boundaries between types rest more on intent than on question text, so the distinctness claim needs qualification. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The carrying object is the QuO model itself: a taxonomy of five CQ types anchored by formal definitions (Definitions 1 to 6) and entity-relationship diagram snippets that specify the mandatory constituents of each type. The key components are the primitives DomainEntity, SubjectDomain, Ontology, foundational-ontology elements, Relationship, Metaproperty, and Metametaproperty, plus the closed sets of relational properties and metaproperties; the definitions use these to give each type a distinct purpose and a distinguishing shape. The model is put to work by annotating a standard development methodology with the CQ types relevant at each stage and by populating ROCQS, a FAIR repository of 438 CQs with examples and templates for each type.
What would settle it
Look at the 438 questions in ROCQS and have two or more ontology engineers, working only from the paper's definitions, assign each question to exactly one type; if agreement is low or many questions are assigned to multiple types, the claimed distinctness is not present in the data.
Extended reading notes
Core claim
The paper's central claim is that the apparent jumble of competency questions in ontology development resolves into a first model with five principal types. Scoping CQs delimit the subject domain and must mention a domain entity and a subject domain; Validating CQs test an ontology's content and must be formalisable and answerable within the ontology's language; Foundational CQs interrogate a domain entity against the vocabulary of a foundational ontology to support alignment; Relationship CQs probe a relationship's arity, elementariness, participants, or relational properties; Metaproperty CQs classify an entity according to metametaproperties such as rigidity, identity, unity, and dependence. Each type is given a formal definition with distinct constituent elements, and the paper argues this accounts for the different uses of CQs documented in the literature. It further demonstrates where each type fits in ontology-development tasks and supplies an annotated repository, ROCQS, containing 438 CQs as evidence and as a resource.
Load-bearing premise
The taxonomy assumes that a competency question's purpose can be read off from its wording and context, and that those purposes sort cleanly into five categories that different ontology engineers would agree on.
Editorial extensions
If this is right
- Developers can choose a CQ type by task: scoping during requirements, validation during implementation, foundational alignment when connecting to upper-level ontologies, and metaproperty or relationship questions during formalisation.
- Validating CQs come with an expressiveness constraint: a question only qualifies if the ontology language can express and answer it, which gives a concrete criterion for dropping unusable CQs.
- Foundational CQs make alignment errors diagnosable: a question is faulty when it asks for distinctions the target foundational ontology does not make.
- Relationship and metaproperty CQs are answered by the modeller through ontological analysis rather than by querying the ontology, changing what tool support for them should look like.
- The ROCQS repository provides a shared, annotated pool of 438 CQs whose type labels can be used to evaluate automated CQ generation and authoring tools.
Reading between the lines
- Because the formal definitions give necessary conditions rather than a decision procedure, the five types are likely to overlap in practice: one and the same question can serve scoping and validation, so the model may be most useful as a multi-label characterisation rather than a partition.
- The same component-based method could be carried over to other requirements-engineering settings: classify any requirement question by purpose and mandatory constituents, then use the type to guide repair; this would test whether the underlying idea generalises beyond ontologies.
- A low-cost experiment the paper does not run is to give the definitions to independent ontology engineers and measure whether they assign repository questions to the same types; a positive result would turn the model into an operational coding scheme.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a model, QuO, for competency questions (CQs) in ontology engineering, claiming that CQs can be discerned and characterised into five principal types: Scoping (SCQ), Validating (VCQ), Foundational (FCQ), Relationship (RCQ), and Metaproperty (MpCQ). Each type is given an informal definition, an EER-style diagram, and a first-order logic formalisation. The paper illustrates the model with a coffee-ontology user story, maps the types onto stages of the NeOn methodology, discusses faulty CQs, and introduces ROCQS, a FAIR repository of 438 annotated CQs. The central claim is that the five types have distinct purposes and identifiable distinct constituent elements, enabling better authoring, validation, and deployment of CQs in ontology development.
Significance. The paper addresses a real gap: although CQs are widely used in ontology engineering, there is little theoretical guidance on their types and constituents. The proposed taxonomy is a useful conceptual contribution if the types can be reliably distinguished, and the accompanying ROCQS repository is a concrete, reusable artifact that can support further empirical work. The paper is also commendable for making its definitions explicit, for attempting formal characterisations, and for connecting the types to methodology stages in NeOn. However, the central claim is currently supported mainly by author-generated examples and a single illustrative user story; there is no inter-annotator study or operational coding procedure. The definitions, as written, do not yet establish that the five types are mutually exclusive or reliably assignable. The significance therefore rests on the plausibility of the taxonomy rather than on demonstrated empirical support, and the formal apparatus contains technical gaps that need to be addressed before it can serve as a reliable basis for classification.
major comments (4)
- [§3.2, Definitions 2 and 3] The definitions of SCQ and VCQ do not distinguish the two types. Definition 2 requires mention of a domain entity and use with an ontology and subject domain; Definition 3 requires mention of a domain entity and validation with an ontology. The only formal difference is the unformalised primitive usage versus validate. The paper itself notes that most SCQs are reused for validation, and the example 'Which animals are endangered?' satisfies both definitions. Thus the claimed 'identifiable distinct constituent elements' are not captured by the formalisations. If the types are distinguished by intent or purpose, that predicate must be made explicit, or the taxonomy should be presented as a purpose-based idealisation rather than a constituent-based classification. This is load-bearing because the central claim is that the types are discernible from their constituents.
- [§3.2, drRCQ formalisation and Definition 3] The first-order logic formalisation of drRCQ contains an unbound variable w: the formula states Ontology(w) and containsVocab(w,y) and containsVocab(w,z) without quantifying w. In addition, the expressiveness constraint in Definition 3, L_v ⊆ L_o, is asserted as sufficient for VCQ answerability without proof; answerability also depends on content coverage and on the query language used. These issues matter because the formalisations are presented as evidence that the types are distinct and can be operationalised.
- [§4.1 and §5] The evaluation does not support the claim that the taxonomy can be reliably discerned. Section 4.1 presents a single user story with examples authored for the purpose, and Section 5 acknowledges that the authors 'needed to design templates and generate CQs for other categories.' There is no inter-annotator study, no coding procedure, and no independent corpus annotation beyond the authors' own repository. The paper should either provide an operational annotation protocol with agreement measures or explicitly reposition the taxonomy as a hypothesis whose empirical validation is future work. Without this, the repository's type labels cannot be taken as evidence of distinct types.
- [§3.2, closing comparison of CQ scope] The paper asserts that 'the ontological CQs—aRCQ, efRCQ, rpRCQ, MpCQ—relate to an entity and the answer to the CQ ought to necessarily hold across all ontologies.' This strong claim is not established by the definitions. For example, a rpRCQ is defined with respect to a particular ontology o and requires that the relational property be expressible in the representation language of o, which makes the answer ontology-relative. Please either justify the universality claim or qualify it, since it is used to divide ontological from domain CQs.
minor comments (5)
- [§3.2, Definition 2] The formalisation contains a typo: 'SQC' should be 'SCQ'.
- [§4.1] The bullet list uses 'FQCs' where 'FCQs' is intended.
- [Throughout] The methodology name is written inconsistently as both 'NeON' and 'NeOn'; please standardise.
- [§4.3] The repository is referenced by a URL only; a DOI or other stable identifier would strengthen the FAIR claim.
- [§4.2] The distinction between VCQs and the listed 'constraint-checking queries' is unclear, since the example 'Are there animals that are both a carnivore and a herbivore?' could also be read as a VCQ; clarifying the criterion would help readers apply the taxonomy.
Circularity Check
SCQ/VCQ distinction rests on unformalised purpose predicates rather than distinct constituents, and the new CQ types are demonstrated with self-authored examples, making the taxonomy's central distinctness claim partially circular.
-
self definitional
[Section 3.2, Definitions 2 and 3 and surrounding text]
"A Validation CQ (VCQ) may sound as if it were the same as, or else a kind of a SCQ, or the set of VCQs for an ontology to be a subset of the set of SCQs, since most SCQs are being reused for validation. There are subtle differences, however, both with respect to intent or purpose (validation versus scoping) and they must be answerable by the ontology."
Definition 2 formalises SCQ as ∀x(SCQ(x) → ∃y,z,w(mention(x,y) ∧ DomainEntity(y) ∧ usage(x,z,w) ∧ Ontology(z) ∧ SubjectDomain(w))) and Definition 3 formalises VCQ as ∀x(VCQ(x) → ∃y,z(mention(x,y) ∧ DomainEntity(y) ∧ validate(x,z) ∧ Ontology(z))). Both require mention of a DomainEntity and an Ontology; the only difference is the unformalised purpose predicates usage versus validate. The paper states most SCQs are reused for validation, so any SCQ sentence also satisfies the VCQ conditions when the intent changes. The claimed 'identifiable distinct constituent elements' are therefore not distinct; type membership is assigned by purpose labels built into the definitions, not by constituents of the question.
-
other
[Section 5 and Section 4.3 (ROCQS); cf. Abstract]
"While there were ample SCQs and VCQs readily available ... we needed to design templates and generate CQs for other categories. ... we created the Repository of Ontology Competency QuestionS (ROCQS), which consists of 438 CQs covering all types described in this paper."
The repository is presented as demonstrating the distinctions among types, but the novel types (FCQ, RCQ, MpCQ) were instantiated by the authors using templates designed from Definitions 4-6. The CQs in the user story are likewise generated by applying the formal definitions. These self-authored instances cannot independently confirm that the categories are real or mutually distinguishable; they are constructed to satisfy the very definitions they are then used to illustrate, so the demonstration is an artefact of construction rather than external validation. Existing SCQ/VCQ data provide some independent grounding, which is why the circularity is partial.
full rationale
The paper proposes a conceptual taxonomy rather than deriving a mathematical or empirical prediction, so most of the content is definitional and not circular in the strict sense. However, the central claim that the five CQ types are distinguished by 'identifiable distinct constituent elements' is contradicted by the paper's own formalisations: the SCQ and VCQ definitions have the same structural constituents and differ only in primitive purpose predicates, and the text explicitly says most SCQs are reused for validation. This means the SCQ/VCQ boundary is imposed by definitional purpose labels rather than discovered in the question text. For the three novel types (FCQ, RCQ, MpCQ), the paper acknowledges it had to generate CQs and templates because such examples were scarce; those self-authored examples are then placed in ROCQS and the coffee user story and presented as demonstrating the distinctions. The repository thus partly validates the model with instances produced from the model, which is a self-definitional loop. The existing dataset of SCQs/VCQs and the alignment-question literature provide some external grounding, so the paper is not wholly circular, but the distinctive part of the contribution rests on self-generated illustration rather than independent evidence. Score 6 reflects partial circularity in the central distinctness claim.
Assumptions & free parameters
free parameters (1)
- Number of CQ types =
5
assumptions (4)
- ad hoc to paper The set of all competency questions for ontologies can be partitioned into five principal types with distinct purposes.
- domain assumption The primitives 'mention' and 'usage' have their usual natural language meaning and can be suitably formalised.
- domain assumption The OntoClean metametaproperties and metaproperties form a closed set usable for MpCQ classification.
- ad hoc to paper The expression-containment constraint L_v ⊆ L_o is sufficient to guarantee that a VCQ is answerable.
invented entities (1)
-
Five-type taxonomy of competency questions (SCQ, VCQ, FCQ, RCQ, MpCQ)
Cite this review
Pith. "Pith review of Discerning and Characterising Types of Competency Questions for Ontologies." pith.science (2026). https://pith.science/paper/UKIX4L3P
@misc{pith2026241213688,
author = {Pith},
title = {Pith review of: Discerning and Characterising Types of Competency Questions for Ontologies},
year = {2026},
howpublished = {\url{https://pith.science/paper/UKIX4L3P}},
note = {Machine review of arXiv:2412.13688}
}
read the original abstract
Competency Questions (CQs) are widely used in ontology development by guiding, among others, the scoping and validation stages. However, very limited guidance exists for formulating CQs and assessing whether they are good CQs, leading to issues such as ambiguity and unusable formulations. To solve this, one requires insight into the nature of CQs for ontologies and their constituent parts, as well as which ones are not. We aim to contribute to such theoretical foundations in this paper, which is informed by analysing questions, their uses, and the myriad of ontology development tasks. This resulted in a first Model for Competency Questions, which comprises five main types of CQs, each with a different purpose: Scoping (SCQ), Validating (VCQ), Foundational (FCQ), Relationship (RCQ), and Metaproperty (MpCQ) questions. This model enhances the clarity of CQs and therewith aims to improve on the effectiveness of CQs in ontology development, thanks to their respective identifiable distinct constituent elements. We illustrate and evaluate them with a user story and demonstrate where which type can be used in ontology development tasks. To foster use and research, we created an annotated repository of 438 CQs, the Repository of Ontology Competency QuestionS (ROCQS), incorporating an existing CQ dataset and new CQs and CQ templates, which further demonstrate distinctions among types of CQs.
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[1]
Alharbi, R., Tamma, V ., Grasso, F., Payne, T.R.: An experiment in retrofitting competency questions for existing ontologies. ArXiv abs/2311.05662 (2023)
arXiv 2023
- [2]
- [3]
-
[4]
Bezerra, C., Freitas, F.: Verifying description logic ontologies based on competency ques- tions and unit testing. In: ONTOBRAS’17. pp. 159–164 (2017)
work page 2017
- [5]
-
[6]
Semantic Web 6(4), 379–386 (2015)
Chiarcos, C., Sukhareva, M.: Olia–ontologies of linguistic annotation. Semantic Web 6(4), 379–386 (2015)
work page 2015
-
[7]
Journal of Information and Data Management 2, 527– 540 (2011)
Fernandes, P., Guizzardi, R., Guizzardi, G.: Using goal modeling to capture competency questions in ontology-based systems. Journal of Information and Data Management 2, 527– 540 (2011)
work page 2011
-
[8]
Transactions on Graph Data and Knowledge2(1), 4:1–4:30 (2024)
Fillottrani, P.R., Keet, C.M.: Logics for Conceptual Data Modelling: A Review. Transactions on Graph Data and Knowledge2(1), 4:1–4:30 (2024). https://doi.org/10.4230/TGDK. 2.1.4
Show all 28 references
-
[9]
Synthese p
Guarino, N., Guizzardi, G.: Processes as variable embodiments. Synthese p. in print (2024)
2024
-
[10]
In: Dieng, R., Corby, O
Guarino, N., Welty, C.: A formal ontology of properties. In: Dieng, R., Corby, O. (eds.) Proc. of EKAW’00. LNCS, vol. 1937, pp. 97–112. Springer (2000)
2000
-
[11]
In: Horn, W
Guarino, N., Welty, C.: Identity, unity, and individuality: towards a formal toolkit for onto- logical analysis. In: Horn, W. (ed.) Proc. of ECAI’00. pp. 219–223. IOS Press (2000)
2000
- [12]
-
[13]
San Francisco: Morgan Kauf- mann Publishers (2001) 16 Keet and Khan
Halpin, T.: Information Modeling and Relational Databases. San Francisco: Morgan Kauf- mann Publishers (2001) 16 Keet and Khan
2001
-
[14]
In: Meersman, R., et al
Halpin, T.A., Curland, M.: Enriched support for ring constraints. In: Meersman, R., et al. (eds.) OTM Workshops 2011. LNCS, vol. 7046, pp. 309–318. Springer (2011)
2011
-
[15]
In: Sack, H., et al
Keet, C.M., Lawrynowicz, A.: Test-driven development of ontologies. In: Sack, H., et al. (eds.) Proc. of ESWC’16. LNCS, vol. 9678, pp. 642–657. Springer, Berlin (2016)
2016
-
[16]
In: Proc
Keet, C.M., Mahlaza, Z., Antia, M.J.: CLaRO: a controlled language for authoring compe- tency questions. In: Proc. of MTSR’19. CCIS, vol. 1057, pp. 3–15. Springer (2019)
2019
-
[17]
In: ten Teije, A., et al
Keet, C.M.: Detecting and revising flaws in OWL object property expressions. In: ten Teije, A., et al. (eds.) Proc. of EKAW’12. LNAI, vol. 7603, pp. 252–266. Springer (2012)
2012
-
[18]
Journal on Biomedical Se- mantics 11(4) (2020)
Keet, C.M.: The African Wildlife Ontology tutorial ontologies. Journal on Biomedical Se- mantics 11(4) (2020)
2020
-
[19]
Law and Human Behavior 25(1), 81–92 (2001)
Lyon, T.D., Saywitz, K.J., Kaplan, D.L., Dorado, J.S.: Reducing maltreated children’s reluc- tance to answer hypothetical oath-taking competency questions. Law and Human Behavior 25(1), 81–92 (2001). https://doi.org/10.1023/A:1005644010134
2001 doi
-
[20]
In: Proc
Monfardini, G.K.Q., Salamon, J.S., Barcellos, M.P.: Use of competency questions in ontol- ogy engineering: A survey. In: Proc. of ER’23. p. 45–64. Springer (2023)
2023
-
[21]
Data in Brief p
Potoniec, J., Wisniewski, D., Lawrynowicz, A., Keet, C.M.: Dataset of ontology competency questions to SPARQL-OWL queries translations. Data in Brief p. in press (2020)
2020
-
[22]
Rebboud, Y ., Tailhardat, L., Lisena, P., Troncy, R.: Can llms generate competency questions? In: Extended Semantic Web Conference (ESWC’24). vol. 659, pp. 227–242. Springer (2024)
2024
-
[23]
NeOn Deliverable D5.4.1, NeOn Project (2008)
Suarez-Figueroa, M.C., et al.: NeOn methodology for building contextualized ontology net- works. NeOn Deliverable D5.4.1, NeOn Project (2008)
2008
-
[24]
In: 13th WS Ontology Matching (OM’18)
Thiéblin, E., Haemmerlé, O., Trojahn, C.: Complex matching based on competency ques- tions for alignment: a first sketch. In: 13th WS Ontology Matching (OM’18). pp. 66–70. CEUR-WS, Monterey, USA (2018)
2018
-
[25]
The Knowl- edge Engineering Review 11(2), 93–136 (1996)
Uschold, M., Gruninger, M.: Ontologies: principles, methods and applications. The Knowl- edge Engineering Review 11(2), 93–136 (1996)
1996
-
[26]
5A How to Arrive at Questions, pp
Watson, L.: chap. 5A How to Arrive at Questions, pp. 176–182. Narr Francke Attempto (2022)
2022
-
[27]
Williams, P.: Resourcing for the future? information technology provision and competency questions for school-based initial teacher education. J. of Inf. Techn. for Teacher Ed. 5(3), 271–282 (1996). https://doi.org/10.1080/0962029960050308
1996 doi
-
[28]
Journal of Web Semantics 59, 100534 (2019)
Wi ´sniewski, D., Potoniec, J., Ławrynowicz, A., Keet, C.M.: Analysis of ontology compe- tency questions and their formalizations in SPARQL-OWL. Journal of Web Semantics 59, 100534 (2019)
2019
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.