Assessing the Capability of Large Language Models for Domain-Specific Ontology Generation

Aldo Gangemi; Andrea Giovanni Nuzzolese; Anna Sofia Lippolis; Eva Blomqvist; Mohammad Javad Saeedizade; Robin Keskisarkka

REVIEW 4 major objections 4 minor 15 references

Assessing the Capability of Large Language Models for Domain-Specific Ontology Generation

T0 review · 4 major / 4 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Reasoning-capable LLMs generate domain-specific ontology drafts from competency questions with consistent quality across six unrelated domains, suggesting LLM-based ontology construction generalizes beyond single-domain settings.

desk verdict Useful multi-domain benchmark for LLM-based ontology generation, but single-run sampling undercuts the 'irrespective of domain' headline claim. read the letter →

arxiv 2504.17402 v1 pith:WHQCVNFS submitted 2025-04-24 cs.AI

Anna Sofia Lippolis , Mohammad Javad Saeedizade , Robin Keskisarkka , Aldo Gangemi , Eva Blomqvist , Andrea Giovanni Nuzzolese This is my paper · ORCID

classification cs.AI

keywords ontologygenerationlargelanguagemodelscompetencyquestionsdomaingeneralizationreasoningfew-shotpromptingknowledgerepresentationengineering

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Ontologies are formal models of a domain's classes, properties, and relationships, and building them by hand is slow and expert-dependent. This paper asks whether two reasoning-capable large language models, DeepSeek and o1-preview, can draft such ontologies directly from natural-language requirements called competency questions, each paired with a short user story. Using 95 curated competency questions across six domains, the paper reports that both models produced Turtle ontology drafts that modelled most questions, with only single-digit numbers of failures and no consistent pattern of one domain being far worse, apart from the Events domain for DeepSeek. If the claim holds, ontology engineering could begin from an automated first draft that a human expert refines, making knowledge modelling faster and less dependent on a single domain's training data.

What carries the argument

The load-bearing piece is the few-shot prompt: an instruction stating the ontology-generation task, a fixed Turtle example ontology paired with one sample competency question and story, and the new question and story as input. Every competency question is processed independently, so each output can be attributed to a single requirement and evaluated per question. Evaluation uses the paper's definition of a modelled competency question—existence of a SPARQL query that retrieves the answer—plus a 'minor issue' category for ontologies one property away from satisfying the question. The consistency claim rests on how these error counts distribute across domains and difficulty levels.

What would settle it

Rerun each of the 95 prompts several times, for example five times at three different temperatures, and tally strict 'not modelled' and 'minor issue' counts per domain; if any domain's error count swings from near zero to several across runs, or a new domain with recently coined terminology shows a large drop, the domain-agnostic consistency claim would be contradicted.

Watch

Extended reading notes

Core claim

The paper's central claim is that ontology generation from competency questions generalizes across domains when the LLM is a reasoning model. In its setup, each competency question is sent to the model in a single API call together with its user story and one fixed few-shot example, and the output is checked against the paper's definition of 'modelled': a SPARQL query must exist that answers the competency question from the generated ontology. A separate 'minor issue' category captures outputs that are one object or data property short of satisfying the question. The reported error counts are small and fairly flat across the six domains, and hard questions are not noticeably worse than easy ones, leading the paper to conclude that the method is domain-agnostic and that earlier single-domain feasibility results extend to new domains.

Load-bearing premise

The comparison assumes that one API call per competency question with default settings is enough to represent each model's capability; if the same prompt can yield different Turtle outputs on reruns, the per-domain error counts and the consistency pattern could change.

Editorial extensions

If this is right

Ontology engineering teams could treat LLM output as a cheap first draft, with human experts repairing missing properties instead of modelling from scratch.
Requirement difficulty may not be the main driver of LLM failure, since hard and easy competency questions produced similar error counts.
Model choice can be tailored to repair cost: o1-preview's errors tend to be minor omissions, while DeepSeek tends to leave more incomplete models.
The approach could be applied to domains that lack existing ontologies, as long as competency questions and user stories can be written.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A stronger test than six domains would be a domain whose terminology is too recent or too niche to appear in pretraining; a performance drop there would show the models are relying on domain familiarity rather than a general ontology-generation skill.
Since each prompt was run only once, the paper has not measured run-to-run variance; repeated sampling could convert the observed consistency into a statistical claim and reveal whether Events is a stable weak spot or noise.
The complementary error profiles suggest a hybrid workflow the paper does not propose: use one model to generate a near-complete draft and the other to supply missing structural elements, or route competency questions by difficulty.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Bottom line: this is a solid, useful capability probe, but the abstract oversells it. The dataset is the real contribution; the 'irrespective of the domain' claim is stronger than the evidence because each CQ was generated once, with default sampling.

The paper evaluates two reasoning LLMs (DeepSeek and o1-preview) on 95 competency questions across six domains, using a few-shot prompt. That is genuinely new: earlier work was single-domain or didn't report per-domain performance. The manual evaluation by two ontology engineers is careful, the error taxonomy is consistent with their prior work, and they are transparent about the Events anomaly and about dataset leakage risk. Code and data are linked. Credit where due: this gives the community a reusable multi-domain benchmark and a clear picture that, on one run, both models look reasonably competent on most domains.

The soft spot is the one the abstract doesn't fully live down. Table 2 reports a single generated ontology per competency question. The models are stochastic at default temperature, so each error count is one draw from an unknown distribution. DeepSeek's Events count of 5/18 error-involved CQs, versus at most 2 elsewhere, is uncomfortably close to what you'd see from sampling noise with n=18; it could also be a real domain-specific weakness. Either way, that single count undermines the categorical 'irrespective of the domain.' The body of the paper acknowledges the Events issue, so the authors are not hiding it, but the abstract and conclusion repeat the strong claim. Also, they don't report inter-annotator agreement for the two engineers; that matters because the minor-issue category is a judgment call.

The rest of the methodology is standard. The pipeline and taxonomies come from their own prior work; that's not a flaw here, since the new thing is the multi-domain evaluation. I'd like repeated runs, confidence intervals, and a softer headline. But those are revisions, not reasons to reject.

This is a paper for the ontology engineering community, especially people building LLM-assisted modeling pipelines. I'd bring it to a reading group with the sampling question on the table. It deserves serious peer review; send it out, ask for repeated sampling and a revised claim.

Referee Report

4 major / 4 minor

Summary. The paper proposes an automated pipeline for domain-specific ontology generation from competency questions (CQs) and user stories using two reasoning-capable LLMs, DeepSeek and o1-preview. The authors introduce a few-shot prompt strategy, build a dataset of 95 CQs across six domains taken from existing ontology projects, and manually evaluate the generated ontologies for whether each CQ is modelled. The central claim is that both models perform consistently across all six domains, indicating that the approach generalizes 'irrespective of the domain'. The paper also reports an error analysis indicating that most errors occur in the Events domain and discusses limitations including dataset leakage and the use of only two LLMs.

Significance. If the central claim were fully supported, this would be a valuable contribution: it is one of the first systematic cross-domain evaluations of reasoning LLMs for CQ-driven ontology generation, it uses externally sourced competency questions rather than self-generated ones, and the authors provide data and code. The per-domain error breakdown is also useful for the community. However, the headline generalization claim currently rests on a small number of single-run outputs, and the reported results themselves contain a visible domain discrepancy for DeepSeek in the Events domain. The contribution is therefore promising but needs additional experimental support and a more carefully qualified interpretation before it can be accepted as stated.

major comments (4)

[Section 5, Table 2] The central claim that performance is 'remarkably consistent across all domains' is not supported by the evidence as reported, because each CQ is sent to the API only once with default hyperparameters and no repeated sampling, confidence intervals, or statistical tests are reported. LLM API outputs are stochastic, and the per-domain error counts are small: for example, DeepSeek has 5/18 errors in Events versus 1 or 2 in most other domains, and o1-preview has 0-2 errors per domain. With a single draw per CQ, the pattern in Table 2 could change substantially on a second run. The paper should either repeat the experiments (e.g., multiple runs per CQ with variance reported and a suitable test for domain differences) or substantially soften the 'irrespective of the domain' claim.
[Table 2 and Section 6] There is an internal inconsistency between the abstract/conclusion and the reported results. The abstract says performance is 'remarkably consistent across all domains, indicating that these methods are capable of generalizing ontology generation tasks irrespective of the domain', while Section 6 states that for DeepSeek 'the scores for the Events domain were noticeably lower'. Table 2 shows DeepSeek's Events error rate is 27.8% (5/18), compared with at most 12.5% in every other domain. Since the paper itself acknowledges a clear exception to cross-domain consistency, the categorical wording in the abstract and conclusion should be revised to reflect the observed Events-specific weakness or the analysis should explain why this exception does not undermine the generalization claim.
[Section 7] The experimental design does not isolate 'domain' from other properties of the requirements, so the claim of domain-agnostic generalization is confounded. The authors note in the error analysis that the Events CQs are 'much shorter' and have 'multiple user stories', and they suggest that longer, more descriptive CQs may aid model performance. Because domain and CQ format vary together across the six datasets, the observed differences (or similarities) cannot be attributed to the domain alone. The paper should either control for CQ length and number of user stories, or explicitly limit the conclusion to 'consistent across the specific CQ sets used here' rather than 'irrespective of the domain'.
[Section 4.1] The evaluation is based on manual annotation by two ontology engineers, but the paper reports no inter-annotator agreement measure (e.g., Cohen's kappa) and no description of how disagreements were resolved. Since the per-domain error counts in Table 2 are the sole quantitative support for the main claim, the reliability of this manual labelling is load-bearing. The authors should report annotation agreement or provide evidence that the labelling procedure is consistent and reproducible.

minor comments (4)

[Section 5] The paper states that several prompt configurations were evaluated and that few-shot outperformed zero-shot, but the detailed results are omitted 'in the interest of conciseness'. Please provide these results in a supplementary appendix or at least report the magnitude of the difference, since the choice of prompt strategy is methodologically important.
[Section 7] The speculation that DeepSeek and o1-preview 'benefit from mutual fine-tuning or shared architectural strengths' is unsupported by the experiments and should be explicitly labelled as speculation or removed.
[Section 4.1] The description of the AquaDiva dataset as 'semi-automatically annotated' is vague; please specify which parts were automated and how, since this affects the reliability of the source CQs.
[General] The GitHub repository link should include a version tag or commit hash so that the prompt, data, and evaluation scripts can be referenced reproducibly.

Circularity Check

0 steps flagged · score 1.0 of 10

No circular derivation; the central evaluation is anchored in external CQs and manual assessment, with only minor inherited self-citations.

full rationale

The paper's central claim—consistent ontology-generation performance across domains—is evaluated against 95 competency questions drawn from four external ontology projects (Onto-DESIDE, Polifonia, WHOW, AquaDiva), and the generated ontologies are manually labelled by ontology engineers as modelled, not modelled, or containing a minor issue. The evaluation criterion is therefore not defined in terms of the LLMs' own outputs, and no fitted parameter is renamed as a prediction. The authors do reuse their own prior work for the generation method and error taxonomy (Lippolis et al. [10]), and they used o1-preview to assign initial easy/hard difficulty labels before manual verification; these are inherited design choices and annotations, not results that are forced by construction. The single-run evaluation with default hyperparameters and the lack of repeated sampling is a statistical robustness limitation, not a circularity, and the paper itself acknowledges dataset-leakage and scope limitations. No equation or definition in the paper reduces the reported cross-domain comparison to its inputs, so no specific circular step can be exhibited. Score 1 reflects the presence of minor self-citations that are not load-bearing to the main empirical claim.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central evaluation rests on no fitted numeric parameters. The few-shot example and prompt wording are design choices rather than fitted values. All inputs are existing competency questions from external ontology projects, and the only new artifacts are the prompts and the evaluation dataset, neither of which introduces a new physical or conceptual entity.

assumptions (4)

domain assumption Competency questions and user stories from six existing projects are sufficient and representative requirements for judging domain-specific ontology generation quality.
Section 4.1 builds the evaluation dataset entirely from these sources; the cross-domain generalization claim depends on this selection representing the space of ontology domains.
domain assumption Manual labelling by two ontology engineers provides accurate ground truth for whether a competency question is modelled.
Section 4.1 'Manual annotations' defines the only outcome measure, but the paper reports no inter-annotator agreement or adjudication, leaving label noise unmeasured.
domain assumption A competency question counts as modelled if a SPARQL query can extract its answer, regardless of the quality of the underlying modelling.
Section 3 defines 'Modelling Competency Question' this way; all accuracy numbers inherit this deliberately narrow definition.
domain assumption A single API call per competency question with default hyperparameters is representative of each model's performance.
Section 5 describes sending each prompt once and saving one Turtle file; no repeated sampling or variance estimate is reported, so stochastic variation is not accounted for.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Assessing the Capability of Large Language Models for Domain-Specific Ontology Generation." pith.science (2026). https://pith.science/paper/WHQCVNFS

@misc{pith2026250417402,
  author       = {Pith},
  title        = {Pith review of: Assessing the Capability of Large Language Models for Domain-Specific Ontology Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WHQCVNFS}},
  note         = {Machine review of arXiv:2504.17402}
}

read the original abstract

Large Language Models (LLMs) have shown significant potential for ontology engineering. However, it is still unclear to what extent they are applicable to the task of domain-specific ontology generation. In this study, we explore the application of LLMs for automated ontology generation and evaluate their performance across different domains. Specifically, we investigate the generalizability of two state-of-the-art LLMs, DeepSeek and o1-preview, both equipped with reasoning capabilities, by generating ontologies from a set of competency questions (CQs) and related user stories. Our experimental setup comprises six distinct domains carried out in existing ontology engineering projects and a total of 95 curated CQs designed to test the models' reasoning for ontology engineering. Our findings show that with both LLMs, the performance of the experiments is remarkably consistent across all domains, indicating that these methods are capable of generalizing ontology generation tasks irrespective of the domain. These results highlight the potential of LLM-based approaches in achieving scalable and domain-agnostic ontology construction and lay the groundwork for further research into enhancing automated reasoning and knowledge representation techniques.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

15 extracted references · 13 canonical work pages

[1]

In: European Semantic Web Conf erence

Algergawy, A., Hamed, H., Thiel, S., König-Ries, B.: Towa rds semantic annota- tion for scientiﬁc datasets. In: European Semantic Web Conf erence. pp. 164–167. Springer (2024)

work page 2024
[2]

In: 15th I nternational Semantic Web Applications and Tools for Healthcare and Life Sciences (SW AT4HCLS 2024) (to appear) (2024) Title Suppressed Due to Excessive Length 9

Alharbi, R., Ahmed, U., Dobriy, D., Łajewska, W., Menotti , L., Saeedizade, M.J., Dumontier, M.: Exploring the role of generative ai in constr ucting knowledge graphs for drug indications with medical context. In: 15th I nternational Semantic Web Applications and Tools for Healthcare and Life Sciences (SW AT4HCLS 2024) (to appear) (2024) Title Suppressed ...

work page 2024
[3]

arXiv preprint arXiv:2407

Benson, C., Sculley, A., Liebers, A., Beverley, J.: My ont ologist: Evaluating bfo- based ai for deﬁnition support. arXiv preprint arXiv:2407. 17657 (2024)

work page 2024
[4]

In: International Semantic Web Conference

de Berardinis, J., Carriero, V.A., Jain, N., Lazzari, N., Meroño-Peñuela, A., Poltronieri, A., Presutti, V.: The polifonia ontology netw ork: Building a semantic backbone for musical heritage. In: International Semantic Web Conference. pp. 302–322. Springer (2023)

work page 2023
[5]

Applied Sciences 15(4), 2146 (2025)

Doumanas, D., Soularidis, A., Spiliotopoulos, D., Vassi lakis, C., Kotis, K.: Fine- tuning large language models for ontology engineering: A co mparative analysis of gpt-4 and mistral. Applied Sciences 15(4), 2146 (2025)

work page 2025
[6]

arXiv preprint arXiv:241 2.02035 (2024)

Fathallah, N., Staab, S., Algergawy, A.: Llms4life: Larg e language models for on- tology learning in life sciences. arXiv preprint arXiv:241 2.02035 (2024)

work page 2024
[7]

Huang, Y., Karabulut, E., Degeler, V.: Large language mod el for ontology learning in drinking water distribution network domain (2024)

work page 2024
[8]

In: European Semantic Web Conference

Lippolis, A.S., Ceriani, M., Zuppiroli, S., Nuzzolese, A .G.: Ontogenia: Ontology generation with metacognitive prompting in large language models. In: European Semantic Web Conference. pp. 259–265. Springer (2024)

work page 2024

Show all 15 references

[9]

Scientiﬁc Data 12(1), 274 (2025)

Lippolis, A.S., Lodi, G., Nuzzolese, A.G.: The water heal th open knowledge graph. Scientiﬁc Data 12(1), 274 (2025)

2025
[10]

arXiv preprint arXiv:2503.05388 (2025)

Lippolis, A.S., Saeedizade, M.J., Keskisärkkä, R., Zup piroli, S., Ceriani, M., Gangemi, A., Blomqvist, E., Nuzzolese, A.G.: Ontology gene ration using large language models. arXiv preprint arXiv:2503.05388 (2025)

2025 arXiv
[11]

M.: Dataset of ontology competency questions to sparql-owl queries translations

Potoniec, J., Wiśniewski, D., Ławrynowicz, A., Keet, C. M.: Dataset of ontology competency questions to sparql-owl queries translations. Data in brief 29, 105098 (2020)

2020
[12]

In: Proc

Presutti, V., Daga, E., Gangemi, A., Blomqvist, E.: extr eme design with content ontology design patterns. In: Proc. Workshop on Ontology Pa tterns. pp. 83–97. CEUR-WS (2009)

2009
[13]

In: European Semantic Web Conference

Saeedizade, M.J., Blomqvist, E.: Navigating ontology d evelopment with large lan- guage models. In: European Semantic Web Conference. pp. 143 –161. Springer (2024)

2024
[14]

Information Processing & Management 62(3), 104042 (2025)

Val-Calvo, M., Egaña Aranguren, M., Mulero-Hernández, J., Almagro- Hernández, G., Deshmukh, P., Bernabé-Díaz, J.A., Espinoza -Arias, P., Sánchez-Fernández, J.L., Mueller, J., Fernández-Breis, J .T.: Ontogenix: Leveraging large language models for enhanced ontology eng ineering...

2025
[15]

prompt.png

Xu, Z., Shen, Z., Zhou, Q., Ristoski, P.: Knowledge graph -enhanced retrieval aug- mented generation for e-commerce (2024) This figure "prompt.png" is available in "png" format from: http://arxiv.org/ps/2504.17402v1

2024 arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.

[1] [1]

In: European Semantic Web Conf erence

Algergawy, A., Hamed, H., Thiel, S., König-Ries, B.: Towa rds semantic annota- tion for scientiﬁc datasets. In: European Semantic Web Conf erence. pp. 164–167. Springer (2024)

work page 2024

[2] [2]

In: 15th I nternational Semantic Web Applications and Tools for Healthcare and Life Sciences (SW AT4HCLS 2024) (to appear) (2024) Title Suppressed Due to Excessive Length 9

Alharbi, R., Ahmed, U., Dobriy, D., Łajewska, W., Menotti , L., Saeedizade, M.J., Dumontier, M.: Exploring the role of generative ai in constr ucting knowledge graphs for drug indications with medical context. In: 15th I nternational Semantic Web Applications and Tools for Healthcare and Life Sciences (SW AT4HCLS 2024) (to appear) (2024) Title Suppressed ...

work page 2024

[3] [3]

arXiv preprint arXiv:2407

Benson, C., Sculley, A., Liebers, A., Beverley, J.: My ont ologist: Evaluating bfo- based ai for deﬁnition support. arXiv preprint arXiv:2407. 17657 (2024)

work page 2024

[4] [4]

In: International Semantic Web Conference

de Berardinis, J., Carriero, V.A., Jain, N., Lazzari, N., Meroño-Peñuela, A., Poltronieri, A., Presutti, V.: The polifonia ontology netw ork: Building a semantic backbone for musical heritage. In: International Semantic Web Conference. pp. 302–322. Springer (2023)

work page 2023

[5] [5]

Applied Sciences 15(4), 2146 (2025)

Doumanas, D., Soularidis, A., Spiliotopoulos, D., Vassi lakis, C., Kotis, K.: Fine- tuning large language models for ontology engineering: A co mparative analysis of gpt-4 and mistral. Applied Sciences 15(4), 2146 (2025)

work page 2025

[6] [6]

arXiv preprint arXiv:241 2.02035 (2024)

Fathallah, N., Staab, S., Algergawy, A.: Llms4life: Larg e language models for on- tology learning in life sciences. arXiv preprint arXiv:241 2.02035 (2024)

work page 2024

[7] [7]

Huang, Y., Karabulut, E., Degeler, V.: Large language mod el for ontology learning in drinking water distribution network domain (2024)

work page 2024

[8] [8]

In: European Semantic Web Conference

Lippolis, A.S., Ceriani, M., Zuppiroli, S., Nuzzolese, A .G.: Ontogenia: Ontology generation with metacognitive prompting in large language models. In: European Semantic Web Conference. pp. 259–265. Springer (2024)

work page 2024