{"id":"769f3f79-8ecc-4129-b08e-09d27edfc432","arxiv_id":"2504.17402","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"DeepSeek and o1-preview both produced correctly modelled ontologies for the large majority of 95 competency questions across six domains, with DeepSeek's errors concentrated in the Events domain.","lead":"The paper tests two reasoning-capable large language models, DeepSeek and OpenAI's o1-preview, by asking them to generate ontology drafts from 95 competency questions across six domains. The reported result is high, consistent modelling accuracy in most domains, which suggests automated ontology drafting can move beyond a single topic.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Single-run sampling with default hyperparameters leaves the 'irrespective of the domain' claim unsupported; DeepSeek's Events-domain error rate (5/18) is a visible point of fragility.","rationale":"The paper is a useful multi-domain evaluation, and I do not doubt the good faith of the authors; the GitHub link with code and data supports reproducibility. My concern is specific to the inference drawn from Table 2. The reader's weakest assumption—one API call per CQ with default hyperparameters—is the right focal point, and I agree with that diagnosis. The lack of repeated runs is especially damaging because the headline claim is about consistency across domains, yet the observed dispersion is small enough (0–2 errors for o1-preview per domain; 0–5 for DeepSeek) that a second run could plausibly change the ordering of domains or remove the Events anomaly. A permutation or mixed-model analysis of repeated runs would settle whether the DeepSeek Events deficit is systematic. I also note that Section 6's statement that excluding minor issues reduces o1-preview's unmodeled CQs from 9 to 8 is difficult to reconcile with the Table 2 layout if the count of minor issues is 8; this ambiguity argues for a careful re-reporting of the counts. None of this invalidates the dataset or the pipeline, but it does mean the abstract's 'irrespective of the domain' should be made conditional on a repeated-run confirmation. Thus the reader's CONDITIONAL verdict stands unchanged.","tokens_in":6907,"tokens_out":13393,"duration_ms":119451,"concrete_test":"Repeat the full protocol from Section 5 (95 CQs, both models, same few-shot prompt and default hyperparameters) for K=5 independent runs. For each run, record the per-domain error counts from Table 2. Then fit a mixed-effects logistic regression with domain as fixed effect and CQ as random effect (or run a permutation test comparing the Events domain against the others) on the pooled data. If the DeepSeek Events error rate remains outside the confidence interval of the other domains in at least 4 of 5 runs, the domain-specificity concern is real and the abstract should be revised; if the pattern shifts substantially across runs, the consistency claim was an artifact of single-run sampling.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (abstract: 'remarkably consistent across all domains... irrespective of the domain') rests on Table 2, which reports one generated ontology per CQ using default hyperparameters (Section 5). The per-domain error counts are small—o1-preview has 0–2 errors per domain, DeepSeek 0–5—and no repeated sampling, confidence interval, or statistical test is reported. At default temperature these API outputs are stochastic, so each count is a single draw from an unknown distribution. The DeepSeek Events total of 5/18 (27.8%) versus not more than 12.5% in all other domains is the clearest risk: it may be sampling noise with n=18, or it may indicate a genuine domain-specific weakness. The paper itself notes the Events anomaly in Sections 6–7. With one run per CQ, the data cannot distinguish 'consistent generalization' from 'high-variance outputs that happened to look similar in this run,' so the categorical 'irrespective of the domain' phrasing is not supported by the evidence as reported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an automated pipeline for domain-specific ontology generation from competency questions (CQs) and user stories using two reasoning-capable LLMs, DeepSeek and o1-preview. The authors introduce a few-shot prompt strategy, build a dataset of 95 CQs across six domains taken from existing ontology projects, and manually evaluate the generated ontologies for whether each CQ is modelled. The central claim is that both models perform consistently across all six domains, indicating that the approach generalizes 'irrespective of the domain'. The paper also reports an error analysis indicating that most errors occur in the Events domain and discusses limitations including dataset leakage and the use of only two LLMs.","tokens_in":7022,"tokens_out":3837,"duration_ms":39217,"significance":"If the central claim were fully supported, this would be a valuable contribution: it is one of the first systematic cross-domain evaluations of reasoning LLMs for CQ-driven ontology generation, it uses externally sourced competency questions rather than self-generated ones, and the authors provide data and code. The per-domain error breakdown is also useful for the community. However, the headline generalization claim currently rests on a small number of single-run outputs, and the reported results themselves contain a visible domain discrepancy for DeepSeek in the Events domain. The contribution is therefore promising but needs additional experimental support and a more carefully qualified interpretation before it can be accepted as stated.","major_comments":[{"comment":"The central claim that performance is 'remarkably consistent across all domains' is not supported by the evidence as reported, because each CQ is sent to the API only once with default hyperparameters and no repeated sampling, confidence intervals, or statistical tests are reported. LLM API outputs are stochastic, and the per-domain error counts are small: for example, DeepSeek has 5/18 errors in Events versus 1 or 2 in most other domains, and o1-preview has 0-2 errors per domain. With a single draw per CQ, the pattern in Table 2 could change substantially on a second run. The paper should either repeat the experiments (e.g., multiple runs per CQ with variance reported and a suitable test for domain differences) or substantially soften the 'irrespective of the domain' claim.","section":"Section 5, Table 2"},{"comment":"There is an internal inconsistency between the abstract/conclusion and the reported results. The abstract says performance is 'remarkably consistent across all domains, indicating that these methods are capable of generalizing ontology generation tasks irrespective of the domain', while Section 6 states that for DeepSeek 'the scores for the Events domain were noticeably lower'. Table 2 shows DeepSeek's Events error rate is 27.8% (5/18), compared with at most 12.5% in every other domain. Since the paper itself acknowledges a clear exception to cross-domain consistency, the categorical wording in the abstract and conclusion should be revised to reflect the observed Events-specific weakness or the analysis should explain why this exception does not undermine the generalization claim.","section":"Table 2 and Section 6"},{"comment":"The experimental design does not isolate 'domain' from other properties of the requirements, so the claim of domain-agnostic generalization is confounded. The authors note in the error analysis that the Events CQs are 'much shorter' and have 'multiple user stories', and they suggest that longer, more descriptive CQs may aid model performance. Because domain and CQ format vary together across the six datasets, the observed differences (or similarities) cannot be attributed to the domain alone. The paper should either control for CQ length and number of user stories, or explicitly limit the conclusion to 'consistent across the specific CQ sets used here' rather than 'irrespective of the domain'.","section":"Section 7"},{"comment":"The evaluation is based on manual annotation by two ontology engineers, but the paper reports no inter-annotator agreement measure (e.g., Cohen's kappa) and no description of how disagreements were resolved. Since the per-domain error counts in Table 2 are the sole quantitative support for the main claim, the reliability of this manual labelling is load-bearing. The authors should report annotation agreement or provide evidence that the labelling procedure is consistent and reproducible.","section":"Section 4.1"}],"minor_comments":[{"comment":"The paper states that several prompt configurations were evaluated and that few-shot outperformed zero-shot, but the detailed results are omitted 'in the interest of conciseness'. Please provide these results in a supplementary appendix or at least report the magnitude of the difference, since the choice of prompt strategy is methodologically important.","section":"Section 5"},{"comment":"The speculation that DeepSeek and o1-preview 'benefit from mutual fine-tuning or shared architectural strengths' is unsupported by the experiments and should be explicitly labelled as speculation or removed.","section":"Section 7"},{"comment":"The description of the AquaDiva dataset as 'semi-automatically annotated' is vague; please specify which parts were automated and how, since this affects the reliability of the source CQs.","section":"Section 4.1"},{"comment":"The GitHub repository link should include a version tag or commit hash so that the prompt, data, and evaluation scripts can be referenced reproducibly.","section":"General"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: this is a solid, useful capability probe, but the abstract oversells it. The dataset is the real contribution; the 'irrespective of the domain' claim is stronger than the evidence because each CQ was generated once, with default sampling.\n\nThe paper evaluates two reasoning LLMs (DeepSeek and o1-preview) on 95 competency questions across six domains, using a few-shot prompt. That is genuinely new: earlier work was single-domain or didn't report per-domain performance. The manual evaluation by two ontology engineers is careful, the error taxonomy is consistent with their prior work, and they are transparent about the Events anomaly and about dataset leakage risk. Code and data are linked. Credit where due: this gives the community a reusable multi-domain benchmark and a clear picture that, on one run, both models look reasonably competent on most domains.\n\nThe soft spot is the one the abstract doesn't fully live down. Table 2 reports a single generated ontology per competency question. The models are stochastic at default temperature, so each error count is one draw from an unknown distribution. DeepSeek's Events count of 5/18 error-involved CQs, versus at most 2 elsewhere, is uncomfortably close to what you'd see from sampling noise with n=18; it could also be a real domain-specific weakness. Either way, that single count undermines the categorical 'irrespective of the domain.' The body of the paper acknowledges the Events issue, so the authors are not hiding it, but the abstract and conclusion repeat the strong claim. Also, they don't report inter-annotator agreement for the two engineers; that matters because the minor-issue category is a judgment call.\n\nThe rest of the methodology is standard. The pipeline and taxonomies come from their own prior work; that's not a flaw here, since the new thing is the multi-domain evaluation. I'd like repeated runs, confidence intervals, and a softer headline. But those are revisions, not reasons to reject.\n\nThis is a paper for the ontology engineering community, especially people building LLM-assisted modeling pipelines. I'd bring it to a reading group with the sampling question on the table. It deserves serious peer review; send it out, ask for repeated sampling and a revised claim.","headline":"Useful multi-domain benchmark for LLM-based ontology generation, but single-run sampling undercuts the 'irrespective of domain' headline claim.","tokens_in":7613,"tokens_out":2515,"would_cite":true,"duration_ms":24977,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Reasoning-capable LLMs generate domain-specific ontology drafts from competency questions with consistent quality across six unrelated domains, suggesting LLM-based ontology construction generalizes beyond single-domain settings.","keywords":["ontology generation","large language models","competency questions","domain generalization","reasoning models","few-shot prompting","knowledge representation","ontology engineering"],"falsifier":"Rerun each of the 95 prompts several times, for example five times at three different temperatures, and tally strict 'not modelled' and 'minor issue' counts per domain; if any domain's error count swings from near zero to several across runs, or a new domain with recently coined terminology shows a large drop, the domain-agnostic consistency claim would be contradicted.","tokens_in":6668,"feed_emoji":"🧠","tokens_out":13865,"duration_ms":121644,"temperature":0.7,"pith_summary":"Ontologies are formal models of a domain's classes, properties, and relationships, and building them by hand is slow and expert-dependent. This paper asks whether two reasoning-capable large language models, DeepSeek and o1-preview, can draft such ontologies directly from natural-language requirements called competency questions, each paired with a short user story. Using 95 curated competency questions across six domains, the paper reports that both models produced Turtle ontology drafts that modelled most questions, with only single-digit numbers of failures and no consistent pattern of one domain being far worse, apart from the Events domain for DeepSeek. If the claim holds, ontology engineering could begin from an automated first draft that a human expert refines, making knowledge modelling faster and less dependent on a single domain's training data.","feed_headline":"LLM-built ontologies hold steady across six domains","feed_subtitle":"DeepSeek and o1-preview answered 95 competency questions with mostly usable Turtle drafts, pointing to domain-agnostic knowledge modelling.","key_machinery":"The load-bearing piece is the few-shot prompt: an instruction stating the ontology-generation task, a fixed Turtle example ontology paired with one sample competency question and story, and the new question and story as input. Every competency question is processed independently, so each output can be attributed to a single requirement and evaluated per question. Evaluation uses the paper's definition of a modelled competency question—existence of a SPARQL query that retrieves the answer—plus a 'minor issue' category for ontologies one property away from satisfying the question. The consistency claim rests on how these error counts distribute across domains and difficulty levels.","core_discovery":"The paper's central claim is that ontology generation from competency questions generalizes across domains when the LLM is a reasoning model. In its setup, each competency question is sent to the model in a single API call together with its user story and one fixed few-shot example, and the output is checked against the paper's definition of 'modelled': a SPARQL query must exist that answers the competency question from the generated ontology. A separate 'minor issue' category captures outputs that are one object or data property short of satisfying the question. The reported error counts are small and fairly flat across the six domains, and hard questions are not noticeably worse than easy ones, leading the paper to conclude that the method is domain-agnostic and that earlier single-domain feasibility results extend to new domains.","pith_inferences":["A stronger test than six domains would be a domain whose terminology is too recent or too niche to appear in pretraining; a performance drop there would show the models are relying on domain familiarity rather than a general ontology-generation skill.","Since each prompt was run only once, the paper has not measured run-to-run variance; repeated sampling could convert the observed consistency into a statistical claim and reveal whether Events is a stable weak spot or noise.","The complementary error profiles suggest a hybrid workflow the paper does not propose: use one model to generate a near-complete draft and the other to supply missing structural elements, or route competency questions by difficulty."],"forward_implications":["Ontology engineering teams could treat LLM output as a cheap first draft, with human experts repairing missing properties instead of modelling from scratch.","Requirement difficulty may not be the main driver of LLM failure, since hard and easy competency questions produced similar error counts.","Model choice can be tailored to repair cost: o1-preview's errors tend to be minor omissions, while DeepSeek tends to leave more incomplete models.","The approach could be applied to domains that lack existing ontologies, as long as competency questions and user stories can be written."],"supporting_citations":[{"why":"Supplies the modelled-competency-question definition, the minor-issue error class, and the Independent Ontology Generation method used for all experiments.","marker":"[10]"},{"why":"Earlier single-domain demonstration that LLMs can draft ontologies from competency questions; this paper extends it across six domains.","marker":"[13]"},{"why":"Earlier prompting study behind the choice of minimal few-shot instructions for reasoning models.","marker":"[8]"},{"why":"Source of the Music and Events competency questions in the evaluation dataset.","marker":"[4]"},{"why":"Source of the Water and Health competency questions.","marker":"[9]"},{"why":"Source of the Microbe Habitat and Carbon and Nitrogen Cycling competency questions.","marker":"[1]"}],"fun_headline_variants":["Reasoning LLMs show steady ontology skills across six domains","Domain-agnostic ontology generation: reasoning LLMs pass six-domain test","DeepSeek and o1-preview build consistent ontologies across domains","LLMs answer 95 ontology questions consistently across six domains","Reasoning models produce domain-agnostic ontologies from user stories"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison assumes that one API call per competency question with default settings is enough to represent each model's capability; if the same prompt can yield different Turtle outputs on reruns, the per-domain error counts and the consistency pattern could change.","fun_headline_variants_meta":{"raw":{"variants":["Reasoning LLMs show steady ontology skills across six domains","Domain-agnostic ontology generation: reasoning LLMs pass six-domain test","DeepSeek and o1-preview build consistent ontologies across domains","LLMs answer 95 ontology questions consistently across six domains","Reasoning models produce domain-agnostic ontologies from user stories"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000639,"raw_usage":{"total_tokens":2917,"prompt_tokens":893,"completion_tokens":2024,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":509,"completion_tokens_details":{"reasoning_tokens":1935}},"tokens_in":509,"tokens_out":2024,"duration_ms":13377,"temperature":1.0,"reasoning_tokens":1935,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:40:34.811728+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun each of the 95 prompts several times, for example five times at three different temperatures, and tally strict 'not modelled' and 'minor issue' counts per domain; if any domain's error count swings from near zero to several across runs, or a new domain with recently coined terminology shows a large drop, the domain-agnostic consistency claim would be contradicted.","supporting_citations":[{"cited_title":"In: European Semantic Web Conference","cited_arxiv_id":null,"evidence_quote":"Earlier single-domain demonstration that LLMs can draft ontologies from competency questions; this paper extends it across six domains."},{"cited_title":"In: European Semantic Web Conference","cited_arxiv_id":null,"evidence_quote":"Earlier prompting study behind the choice of minimal few-shot instructions for reasoning models."},{"cited_title":"In: International Semantic Web Conference","cited_arxiv_id":null,"evidence_quote":"Source of the Music and Events competency questions in the evaluation dataset."},{"cited_title":"Scientific Data 12(1), 274 (2025)","cited_arxiv_id":null,"evidence_quote":"Source of the Water and Health competency questions."},{"cited_title":"In: European Semantic Web Conf erence","cited_arxiv_id":null,"evidence_quote":"Source of the Microbe Habitat and Carbon and Nitrogen Cycling competency questions."}],"review_version":1}