{"id":"00d08cff-b814-4cde-b58e-c7a520cc4ef3","arxiv_id":"2412.20942","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"An LLM pipeline that generates competency questions from documents, aligns extracted relations to Wikidata properties, and outputs RDF triples grounded in the resulting ontology achieves competitive partial-F1 on Wiki-NRE, SciERC, and WebNLG.","lead":"The paper builds knowledge graphs from documents by first asking an LLM to invent competency questions, then aligning the implied relations with Wikidata properties, and finally generating RDF triples grounded in that ontology. A generalist might read it to see whether an LLM pipeline can produce reusable, schema-aligned knowledge graphs with little manual ontology engineering.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Target-schema-constrained evaluation is confounded: the method is given the test schema while LLM baselines are not, so the headline gains may reflect schema provisioning, not the ontology-grounded pipeline.","rationale":"The reader's weakest assumption identifies CQ-answer coverage as the main limitation. That is a real concern: the pipeline only extracts knowledge surfaced by generated competency questions, so recall is capped by CQ coverage, and the paper never measures the ceiling. However, the headline claim is specifically about exceeding baselines in the target-schema-constrained setting. The more immediately load-bearing issue is that this setting hands the method the test relation schema, while the cited LLM baselines were not demonstrably given the same schema. The extremely low SciERC baseline (0.07) and the paper's own note about parsing output KG with an RDF parser suggest the baselines were at a systematic disadvantage. If a fair baseline with the schema provided still loses, then the method's contribution is validated. If not, the claimed improvement is an artifact of evaluation asymmetry. This does not change the overall verdict—the paper still needs substantial additional evidence and controls—but it sharpens the condition: the evaluation must include a schema-controlled LLM baseline. The CQ-coverage concern remains important for interpreting the method's practical completeness, but it is secondary to whether the empirical superiority claim is real.","tokens_in":15585,"tokens_out":5398,"duration_ms":54733,"concrete_test":"Run a direct LLM extraction baseline (same model, e.g., Mistral-7B-instruct) with the target schema explicitly provided in the prompt: list the relation names/descriptions from the test set (the same information the proposed pipeline receives in the constrained setting), instruct the model to extract triples using only those relations, and evaluate with the same RDF-parsing and partial-F1 script used for the proposed method. If this baseline matches or exceeds the proposed method's F1 (0.66/0.71 on Wiki-NRE; 0.73/0.77 on SciERC), the central claim of superiority is not supported. If it falls short, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline result (Section 4.2) is that the proposed pipeline 'exceeds all baseline under target schema constrained setting.' In this setting, Section 4.1 says the authors 'match all relation types in test sets to its closest equivalent in Wikidata and constrict ontology to the relation universe in test set.' Thus the ontology used for KG generation is built directly from the gold relation inventory. Fine-tuned baselines (GenIE, PL-Marker, ReGen) are trained on the training split with the same schema, so they are comparable. But the LLM baselines are cited from prior work ([19] for Wiki-NRE/WebNLG, [3] for SciERC); there is no evidence that those prompts were provided with the target relation schema, and the SciERC score of 0.07 (vs. proposed 0.73) strongly suggests the baseline LLM was not given the relation labels or was evaluated without the authors' RDF-parsing postprocessing. Without a controlled baseline that receives the same schema and the same output parsing, the comparison is confounded: the improvement could come entirely from giving the LLM the answer key for relation types rather than from the ontology-grounded CQ pipeline. This is the crux of the 'ontology-grounded' claim, because the constrained setting is the one where the method wins.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an ontology-grounded LLM pipeline for knowledge graph construction. The pipeline first generates competency questions (CQs) from input documents and answers them, then extracts relations from the CQs, matches those relations to Wikidata properties via embedding similarity plus an LLM vetting step, formats the result as an OWL ontology, and finally constructs RDF triples from the (CQ, answer) pairs constrained by that ontology. Experiments on Wiki-NRE, SciERC, and WebNLG report partial F1 scores, with the best scores obtained under a 'target schema constrained' setting in which the ontology is restricted to the relation universe of the test set. The paper claims that the proposed approach exceeds all baselines on Wiki-NRE and SciERC under that setting and remains competitive on WebNLG, while the 'no schema constraint' setting yields lower scores but captures ontology beyond the test schema.","tokens_in":15883,"tokens_out":4138,"duration_ms":39439,"significance":"If the central claim were cleanly supported, the contribution would be valuable: a zero-shot, fine-tuning-free KG construction pipeline that produces Wikidata-interoperable OWL ontologies and RDF output, with a natural interpretability story via competency questions. The paper also has concrete strengths: it evaluates on three datasets, includes a 7B open model and a frontier model, ships prompts in the appendix, and uses RDF parsing for its own output. However, the evaluation as presented is confounded by schema provisioning. The headline comparison gives the proposed method the gold relation universe while LLM baselines are cited from prior work and are not given the same schema, so the reported gains cannot be attributed to the ontology-grounded pipeline rather than to providing the answer key. The no-schema setting is more honest but underperforms fine-tuned baselines, and the paper does not measure recall against full document content. These issues are load-bearing for the central claim and require substantial experimental rework.","major_comments":[{"comment":"The target-schema-constrained setting is confounded. The authors state that in this setting they 'match all relation types in test sets to its closest equivalent in Wikidata and constrict ontology to the relation universe in test set.' Thus the proposed pipeline is given the gold relation inventory during ontology construction, while the LLM baselines from [19] and [3] are taken from prior work with no evidence that those prompts included the target schema. The headline result in Section 4.2 ('exceeds all baseline under target schema constrained setting on Wiki-NRE and SciERC') may therefore reflect schema provisioning rather than the ontology-grounded pipeline. This is the crux of the paper's claim, because the constrained setting is the only one where the method beats all baselines. I request a controlled comparison where a direct LLM baseline is given the same relation universe (or the same ontology) and the same RDF-parsing postprocessing, plus an ablation that removes the schema constraint while keeping all other components identical.","section":"Section 4.1, Table 1"},{"comment":"The SciERC LLM baseline of 0.07 is implausibly low relative to the proposed method's 0.73, and the paper itself notes it is 'highly unlikely that Mistral-7B poses an advantage over an earlier version of GPT-4' when interpreting this result. This gap strongly suggests that the baseline was not given the target relation labels or was evaluated without comparable output postprocessing. Please verify the baseline prompt setup, the triple extraction/parsing procedure, and the exact evaluation script used for the baseline. Without this, the SciERC comparison is not interpretable.","section":"Table 1, Section 4.1 (SciERC)"},{"comment":"The no-schema constraint setting is described as 'close to real-life applications when processing documents with unknown schema,' but this is also the setting that corresponds to the paper's stated goal of autonomous ontology discovery. In this setting, Mistral scores 0.60/0.58/0.68 on Wiki-NRE/SciERC/WebNLG, below the fine-tuned baselines. The paper should either reposition its central claim around the honest setting or provide additional analysis (e.g., precision/recall breakdown, oracle upper bounds) to show that the lower F1 is acceptable given the expanded ontology. In addition, because the final KG is built only from (CQ, answer) pairs (Sections 3.1 and 3.4), any fact not surfaced by a generated CQ is absent; recall against the full document content is never measured. This is a completeness limitation that should be quantified.","section":"Sections 3.1, 3.4, and 4.1 (no-schema setting)"},{"comment":"The statement 'For property conjunction, evaluate for, compare, feature of in SciERC, we select the closest properties proposed by LLM based on our subjective opinion' introduces manual, subjective alignment of the generated ontology to the gold schema. This is a form of supervision toward the test labels and directly affects the SciERC result. It should be reported as a free parameter, and ideally replaced with an automatic selection rule (or at least the manual choices should be documented and the sensitivity of the results to alternative choices reported).","section":"Section 4.1 (SciERC property selection)"},{"comment":"All experiments are reported as a single run with no error bars, no standard deviations, and no significance tests. The differences between the proposed method and some baselines are small (e.g., Wiki-NRE 0.66 vs. 0.647; WebNLG 0.74 vs. 0.728), so it is not clear that these differences are meaningful without variance estimates. Please report multiple runs (or bootstrap/confidence intervals) and, where feasible, use the same evaluation script for all systems.","section":"Section 4.1 (experimental protocol)"}],"minor_comments":[{"comment":"Typographical errors: 'As mentiond in Introduction' should be 'As mentioned in Introduction'; 'canocalization' should be 'canonicalization'; 'grouding' should be 'grounding'.","section":"Section 2"},{"comment":"The phrase 'constrict ontology' should be 'constrain ontology'.","section":"Section 4.1"},{"comment":"The sentence 'Our proposed approach exceeds all baseline under target schema constrained setting on Wiki-NRE and SciERC datasets' is imprecise because Table 1 shows that on WebNLG the proposed method (0.74) does not exceed the fine-tuned baseline (0.767). Please clarify the claim to match the table.","section":"Section 4.2"},{"comment":"The note that the official WebNLG annotation only contains place-of-birth and nationality triples for the sample document is useful, but it appears only in an appendix and only for one example. Please move this caveat into the main evaluation section and clarify how the partial F1 scoring handles such cases across the dataset.","section":"Appendix A"},{"comment":"The prompt in C.3 contains a duplicated phrase ('all relationships and concepts concepts mentioned'), and the prompt in C.5 has 'clousre' instead of 'closure'. These should be fixed.","section":"Appendix C"},{"comment":"The paper states 'All prompts are reused across all datasets,' but the ontology used in the KG-generation prompt necessarily differs between the constrained and unconstrained settings. Please clarify which parts are held fixed and which vary.","section":"Appendix C"},{"comment":"Reference [14] is cited as 'AutoKG' in the text but the bibliography entry title is 'AutoKG: Efficient Automated Knowledge Graph Generation for Language Models', which is a different work from reference [13] (also titled 'AutoKG'). Please disambiguate these citations and ensure the correct reference is cited in Section 4.3.3.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper is a workshop-style submission whose central claim is currently undercut by a schema-provisioning confound. The fix is feasible within the manuscript's scope: run a controlled LLM baseline with the same schema and the same RDF parsing, report variance, and reposition the contribution around the no-schema setting or clearly delimit the constrained setting as an oracle/upper-bound scenario. The authors should also disclose the manual SciERC property selection prominently. If these experiments confirm the pattern, the paper could become a solid contribution; as is, the evaluation does not support the headline."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The genuinely new piece here is using competency questions to scope the ontology, aligning extracted relations to Wikidata, then grounding triple generation on that ontology. The two evaluation settings (constrained vs. no-schema) are a good idea, and the authors are honest about the trade-off: the no-schema version discovers a richer ontology but scores lower against dataset schemas. That framing is useful.\n\nThe method is clearly described, and the prompts are included, so a reader could reproduce it if the data were available. The appendix sample KG shows they're at least producing RDF that parses.\n\nBut the headline claim doesn't survive scrutiny. In the 'target schema constrained' setting, the ontology is built directly from the test set's relation universe (Section 4.1). The LLM baselines are taken from prior work and were not given that same relation list. So the comparison is not apples to apples: your pipeline gets the answer key for relation types, the baselines don't. That's especially visible in SciERC, where the cited GPT-4 baseline scores 0.07. That number is implausibly low for any reasonably prompted LLM, and it suggests either a different prompting setup or no schema information. Without a controlled baseline that receives the same schema and same output parsing, the 'we exceed all baselines' claim is unsupported.\n\nThere are smaller issues: the SciERC property matching includes manual selection (subjective, as they admit), there are no error bars, and no code is released. Also, the CQ generation is the bottleneck—since the KG is built only from (CQ, answer) pairs, any fact not surfaced by a generated question is lost. Recall against full document content is never measured, so the pipeline's completeness is unknown. The 'Wikidata interoperability' claim is also a bit strong: properties map to Wikidata P-ids, but entities are rendered as labels, not canonical Q-ids, so real knowledge base expansion would need more work.\n\nOverall, this is a plausible pipeline and a reasonable workshop paper, but the evaluation as presented doesn't prove the ontology grounding helps. The no-schema results are more honest and they're not better than a fine-tuned extractor on WebNLG. That said, the approach is worth engaging with: a controlled comparison would be a meaningful next step. I'd send it to review if it came in, but I'd ask for the baseline parity fix and error bars before accepting.","headline":"Plausible pipeline, but the headline comparison is confounded by schema provisioning; needs a controlled baseline and error bars before the core claim is credible.","tokens_in":16363,"tokens_out":2722,"would_cite":false,"duration_ms":26257,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that grounding an LLM's knowledge-graph construction in a Wikidata-aligned ontology, authored by the model itself from competency questions, produces KGs that beat fine-tuned baselines on Wiki-NRE and SciERC while…","keywords":["Knowledge Graph Construction","Ontology Grounding","Wikidata Schema","Competency Questions","Large Language Models","Relation Extraction","Interpretable AI","RDF"],"falsifier":"Measure recall against the full gold standard rather than only the target-schema triples: run the pipeline on the full Wiki-NRE or SciERC test sets and compare the complete set of generated triples (or the set of facts in the document) against all gold triples for those documents. If the recall of the target-schema triples is high but overall document-level recall is low, then the competency-question step is the bottleneck; a stronger test would be to have human experts list all relations in a sample of documents and count how many of them the generated competency questions cover.","tokens_in":15392,"feed_emoji":"🕸️","tokens_out":5285,"duration_ms":49005,"temperature":0.7,"pith_summary":"This paper is trying to establish that a large language model can build high-quality knowledge graphs by first authoring its own ontology from competency questions, aligning that ontology to Wikidata properties, and then grounding all triple generation on that ontology. The claim is that this grounding, rather than larger models or fine-tuning, is what makes the output consistent, interpretable, and competitive with specialized extractors. On the Wiki-NRE and SciERC benchmarks the constrained pipeline exceeds both fine-tuned baselines and earlier LLM baselines, while staying competitive on WebNLG. The authors also argue that the resulting Wikidata-conformant RDF triples are directly usable downstream and auditable, which they present as a path toward interpretable question answering over proprietary document sets.","feed_headline":"LLM knowledge graphs beat fine-tuned rivals with Wikidata grounding","feed_subtitle":"Grounding triple generation on an authored, Wikidata-aligned ontology lifts scores on two benchmarks.","key_machinery":"The load-bearing object is the authored ontology, a set of Wikidata-aligned properties derived from competency questions. The pipeline runs four stages: (1) an LLM writes up to three competency questions per document and answers them using only the document; (2) a relation-extraction prompt turns the questions into candidate properties with usage descriptions, and a sentence-embedding search against a pre-filtered Wikidata property list retrieves the top candidate, which a second LLM vetting step accepts or rejects; (3) the accepted properties are formatted into an OWL ontology, copying Wikidata descriptions, domains and ranges for matched properties; (4) the KG generation prompt is restricted to this ontology only. The ontology is what converts open-ended triple extraction into a constrained generation task and gives the output a Wikidata-compatible semantics.","core_discovery":"The central claim is that ontology grounding—not scale or fine-tuning—is what enables an LLM to construct competitive knowledge graphs in a schema-constrained setting. The pipeline first asks the LLM to write competency questions from each document, extracts candidate relations from those questions, and matches each candidate to a Wikidata property via embedding similarity followed by an LLM veto. The accepted properties are formatted into an OWL ontology, and the KG generation step is forbidden from using any property outside that ontology. With a 7-billion-parameter instruction-tuned model this achieves partial F1 of 0.66 and 0.73 on Wiki-NRE and SciERC, and 0.71 and 0.77 with a stronger proprietary model, beating the non-LLM baselines (0.484 and 0.532) and the LLM baselines (0.647 and 0.07) under the target-schema constraint. Without the constraint, scores drop to 0.60 and 0.58, which the paper explains by the model discovering a richer ontology than the test annotations cover.","pith_inferences":["Because the final KG is built only from (CQ, answer) pairs, the method's completeness is bounded by the competency-question generation step; increasing the number of questions per document or generating them hierarchically could raise recall but would also raise inference cost, so the pipeline's advantage may shrink as documents grow.","The ontology-matching step inherits the weaknesses of embedding similarity: relations that are semantically close but structurally different in Wikidata, such as inverse properties or subproperties, may be missed or conflated even with the LLM veto, so a validation pass against a gold property list would quantify this risk.","A testable extension is to compare the coverage of generated competency questions against a fixed set of expert-authored questions per domain; this would isolate whether the CQ stage or the later triple-extraction stage is the true recall bottleneck.","The claimed interpretability rests on the ontology being small enough to audit; as the property list grows, the interpretability advantage over black-box extraction may weaken."],"forward_implications":["The pipeline produces KGs that parse as RDF triples, so they can be consumed by standard RDF tools without bespoke post-processing.","The target-schema-constrained configuration matches or beats fine-tuned extractors on Wiki-NRE and SciERC, suggesting zero-shot LLM construction can be competitive without task-specific training.","Grounding on Wikidata properties gives the output a path to interoperability, allowing the KG to be merged with or audited against Wikidata.","Users can submit new competency questions to expand the ontology incrementally, providing a human-in-the-loop way to refine schema scope.","The no-schema-constraint configuration trades a small F1 drop for broader ontology coverage, which the paper positions as useful when mining documents of unknown schema."],"supporting_citations":[{"why":"Provides the Wiki-NRE benchmark whose test subset is used for evaluation.","marker":"[16]"},{"why":"Provides the SciERC benchmark with relation types not native to Wikidata.","marker":"[17]"},{"why":"Defines the WebNLG test set and the partial F1 evaluation standard.","marker":"[18]"},{"why":"Supplies the 1000-sample Wiki-NRE test subset, the split, and the LLM baseline numbers reused here.","marker":"[19]"},{"why":"Used as the LLM baseline for SciERC and as background for LLM-based KG construction.","marker":"[3]"},{"why":"The 7-billion-parameter instruction-tuned model used in the main experiments.","marker":"[21]"},{"why":"The sentence embedding model used for property matching.","marker":"[22]"},{"why":"Cited to justify the observation that LLMs recall Wikidata knowledge during ontology generation.","marker":"[15]"}],"fun_headline_variants":["LLM builds better knowledge graphs when ontology-grounded in Wikidata","Wikidata-constrained ontology lifts LLM KG scores on two benchmarks","Ontology grounding, not model size, key to LLM knowledge graphs","LLMs craft competitive knowledge graphs via Wikidata-aligned ontology"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The pipeline assumes that the generated competency questions, capped at three per document in the experiments, surface every fact that ought to be in the knowledge graph, so any fact not elicited by a question is absent from the final KG.","fun_headline_variants_meta":{"raw":{"variants":["LLM builds better knowledge graphs when ontology-grounded in Wikidata","Wikidata-constrained ontology lifts LLM KG scores on two benchmarks","Ontology grounding, not model size, key to LLM knowledge graphs","LLMs craft competitive knowledge graphs via Wikidata-aligned ontology"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000923,"raw_usage":{"total_tokens":3923,"prompt_tokens":877,"completion_tokens":3046,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":493,"completion_tokens_details":{"reasoning_tokens":2982}},"tokens_in":493,"tokens_out":3046,"duration_ms":22255,"temperature":1.0,"reasoning_tokens":2982,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T23:06:02.183855+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure recall against the full gold standard rather than only the target-schema triples: run the pipeline on the full Wiki-NRE or SciERC test sets and compare the complete set of generated triples (or the set of facts in the document) against all gold triples for those documents. If the recall of the target-schema triples is high but overall document-level recall is low, then the competency-question step is the bottleneck; a stronger test would be to have human experts list all relations in a sample of documents and count how many of them the generated competency questions cover.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the SciERC benchmark with relation types not native to Wikidata."},{"cited_title":"Castro Ferreira, C","cited_arxiv_id":null,"evidence_quote":"Defines the WebNLG test set and the partial F1 evaluation standard."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The sentence embedding model used for property matching."}],"review_version":1}