{"id":"5f66c913-90df-40ac-8fc2-5b9b82286bba","arxiv_id":"2509.04696","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A production system using ontology-guided prompts and two LLM stages extracted 19 million high-confidence facts from Wikipedia with 98.8% reported precision.","lead":"ODKE+ is a software system that reads Wikipedia pages and turns them into structured facts for a knowledge graph, reporting 19 million facts added with 98.8% accuracy. It matters because keeping knowledge graphs fresh is normally done by hand, and this shows an automated LLM pipeline can do it at internet scale.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Grounder LLM is unvalidated as a filter; the 98.8% precision and 35% hallucination-reduction claims are unsupported without a staged evaluation against human judgments.","rationale":"The reader's conditional verdict rightly identifies the lack of independent validation for the Grounder and the human audit as the weakest point. My analysis agrees: the paper's central quantitative claims depend on the Grounder's binary judgments being both accurate and unbiased, yet no evidence is provided for that assumption. The 98.8% precision figure is particularly load-bearing because it is presented as the system's headline achievement and is attributed to the combination of grounding and corroboration. However, the paper does not decompose the contribution of each component with human-labeled data. The Grounder could be accepting many false positives that the Corroborator later filters, or it could be rejecting many true positives, inflating precision while harming coverage. The audit protocol is also under-specified: no sampling details, no inter-annotator agreement, no confidence intervals. I also note a second, related issue: the Grounder checks textual support in the retrieved passage, not alignment with the real world; this is a semantic limitation, but it is secondary to the missing validation. The paper has real strengths: a clear architecture, production deployment, and plausible operational metrics; the authors are not claiming formal verification. The concern is not that the system is dishonest, but that the evidence as presented is insufficient to accept the quantitative precision claims. Therefore, the conditional verdict should stand, and the concrete staged evaluation would settle the issue.","tokens_in":8435,"tokens_out":4750,"duration_ms":48848,"concrete_test":"Run a staged evaluation on a random sample of 1,000 Wikipedia pages: (1) collect raw LLM extractions; (2) pass through the Grounder, logging both accepted and rejected triples; (3) pass survivors through the Corroborator. Have three trained annotators independently label each triple as supported or unsupported (compute Cohen's kappa). Report precision at each stage and the Grounder's false-accept and false-reject rates. If precision at stage (2) is not significantly above stage (1), or if the Grounder accepts >10% unsupported triples, the claimed 35% hallucination reduction does not hold; if the final precision estimate's confidence interval includes values below 95%, the 98.8% claim is not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline precision (98.8%, Table 2) is only as trustworthy as the two verification layers that produce it. Section 3.3.1 introduces the Grounder as a second-LLM True/False filter, but the paper reports no precision/recall of this filter against human judgments, no calibration, and no inter-annotator agreement for the human audits that define 'correct' (Section 3.7). Consequently, the stated causal chain—'grounding reduced hallucinated extractions by 35%; corroboration improved precision from 91% to 98.8%'—is unsupported: the 35% could be due to rejecting easy-to-detect errors, and the 98.8% could reflect the Corroborator's scoring rather than the Grounder. Moreover, the Grounder's definition of 'grounded' is textual support in the retrieved passage, not factual truth; a wrong statement in a passage (e.g., a vandalized infobox) would be accepted. Without either an external evaluation of the Grounder or a staged precision measurement, the central quantitative claims are unfalsifiable from the preprint.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper describes ODKE+, a production knowledge-graph extraction pipeline that combines an extraction initiator, evidence retriever, pattern- and LLM-based extractors, an LLM-based grounding verifier, and a corroboration/ranking stage. The authors report deploying the system since May 2025, ingesting 19 million high-confidence facts across 195 predicates, with 98.8% precision, a 35% reduction in hallucinated extractions from grounding, and a 48% overlap with a third-party KG. The central claim is that ontology-guided prompting together with a grounding verifier and corroboration can make LLM-based open-domain knowledge extraction trustworthy at web scale.","tokens_in":8714,"tokens_out":2359,"duration_ms":25244,"significance":"If the reported production metrics are methodologically sound, the paper would be a valuable industrial demonstration that LLM-based knowledge extraction can be deployed with high precision on a large scale. The system's design is internally coherent, and the inclusion of a dynamic ontology-snippet generation pipeline, a lightweight second-LLM grounder, and a corroboration stage are concrete engineering contributions. The paper also presents a favorable comparison of update lag and coverage against prior pipelines. However, the quantitative claims rely almost entirely on internal production monitoring with no released artifacts, no public evaluation protocol, and no detailed error analysis, which limits the paper's current value to the research community as a verifiable scientific contribution.","major_comments":[{"comment":"The headline precision values (98.8%, 91% raw LLM, 95%+ audit accuracy) are presented without any evaluation protocol. Section 3.7 mentions a weekly audit of approximately 2,000 randomly sampled triples, but does not specify the sampling frame, the audit instructions, the number of auditors, or inter-annotator agreement. Without these details, 98.8% is not a statistically defensible claim. The paper should report confidence intervals, sample sizes per metric, and a precise definition of 'correct' used by the auditors.","section":"Section 4.1 / Table 2"},{"comment":"The Grounder is a load-bearing component: the claim that grounding reduced hallucinated extractions by 35% and that corroboration improved precision from 91% to 98.8% depends on the Grounder's True/False judgments being accurate. However, the paper reports no evaluation of the Grounder against human-labeled ground truth, no precision/recall, and no calibration. A staged evaluation—e.g., measuring precision of (i) raw LLM extractions, (ii) post-grounding, and (iii) post-corroboration on the same audited sample—is necessary to support the attribution. The current wording makes the causal chain unfalsifiable.","section":"Section 3.3.1 / Section 4.1"},{"comment":"The '48% overlap with third-party KG' metric is undefined. The paper does not name the third-party KG, describe how overlap was computed (exact triple match? entity+predicate? normalized values?), or specify the evaluation timeframe. Without this, the coverage claim cannot be interpreted or reproduced. The authors should state the KG, matching criterion, and the denominator (e.g., all third-party KG facts for the same entity set? facts present in either KG?).","section":"Section 4.2 / Table 2"},{"comment":"The freshness claim ('facts appear roughly 50 days earlier than legacy KG workflows') lacks a comparison baseline. What is a 'legacy KG workflow' in this context? Is the delay measured from first Wikipedia edit to KG ingestion, or from source publication? The time-lag analysis in Section 3.1 (69 days) uses Wikipedia and Wikidata, but the 50-day improvement is not connected to any measured baseline. Please define both endpoints and the entity/predicate subset used for the comparison.","section":"Section 4.1"}],"minor_comments":[{"comment":"Typos: 'it's hight' should be 'its height'; the example sentence about Taylor Swift is missing a verb ('The Initiator detects recent changes on her Wikipedia page.' should be '...changes to her Wikipedia page.').","section":"Section 3, Example"},{"comment":"The prompts are illustrative but the model names, versions, decoding temperatures, and prompt template versions are not specified. For reproducibility, please identify the LLMs used for extraction and grounding, or state if they are not disclosed due to proprietary constraints.","section":"Appendix A / B"},{"comment":"The AutoML-based scoring (H2O) is mentioned but no feature list, training data, or evaluation of the learned ranker is provided. It is unclear how much of the precision gain is due to the AutoML scorer versus the rule-based heuristics.","section":"Section 3.4"},{"comment":"The side-by-side evaluation result ('prefer ODKE+ in roughly two-thirds of cases') is reported without the number of queries, the number of annotators, or the stratification details. This makes the preference claim difficult to assess.","section":"Section 3.7"}],"recommendation":"major_revision","confidential_remarks":"The reader's report and stress-test note are aligned with my own reading. The system description is compelling, but the missing evaluation protocol is the central obstacle: the paper currently reads more as a production engineering report than as a verifiable research contribution. With a detailed evaluation section, the paper could become acceptable. I would not recommend rejection because the design is plausible and the claimed deployment is specific; however, the authors must disclose enough methodology to make the precision and component-impact claims testable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"ODKE+ is a straightforward production-systems paper: Apple's third iteration of an ontology-guided LLM knowledge-extraction pipeline, deployed since May 2025. The genuinely new pieces are dynamic ontology snippet generation (ranking predicates per entity type and enriching them into prompt templates), a second-LLM grounding filter, and expansion to 195 predicates. The paper is honest that it builds on ODKE v1/v2, and the v1/v2/v3 comparison table is useful.\n\nThe system description is clear and the architecture is coherent. The ontology snippet pipeline is a real engineering contribution: instead of throwing the whole schema at the LLM, they generate entity-type-specific prompts with canonical labels, ranges, units, and qualifiers, and refresh them periodically. That addresses a practical cost and token-efficiency problem in open-domain extraction. The Grounder as a lightweight verifier is a reasonable design choice; the corroboration step with H2O AutoML is standard but well explained.\n\nThe soft spots are all in the evidence for the headline numbers. The 98.8% precision, 35% reduction in hallucinated extractions, and 91% to 98.8% improvement are based on internal weekly audits of about 2,000 triples. The paper gives no sampling protocol, no inter-annotator agreement, no definition of the third-party KG used for the 48% overlap, no error bars, and no released data or code. The Grounder itself is never validated against human judgment, so the causal claim—grounding reduces hallucinations by 35%, corroboration boosts the rest—is an internal production telemetry number, not a demonstrated result. The stress-test note is right: the 35% could be rejecting easy-to-catch errors, and the 98.8% could be driven more by the corroborator's scoring than by the grounder. That doesn't mean the system doesn't work; it means the preprint doesn't give a reader enough to verify it.\n\nThe 48% overlap is also under-specified. And the '50 days earlier' freshness figure is plausible but unmeasured from the outside.\n\nWho is this for? Practitioners building KG ingestion pipelines. The paper is not a scientific breakthrough, but it is a well-described, honest system report with production numbers that are plausible. It deserves peer review—a serious referee should push the authors to add an evaluation section with a clear audit definition, a grounder error analysis, and a per-predicate precision table. Without that, the paper is a demo with impressive-looking metrics; with that, it could be a reference for deploying LLM extraction responsibly. I'd recommend acceptance after major revision, conditional on making the evaluation reproducible.","headline":"A coherent production system report from Apple's KG team, but the headline metrics rest on internal audits with no released protocol—treat 98.8% precision as a claim, not a verified result.","tokens_in":9235,"tokens_out":3567,"would_cite":true,"duration_ms":32848,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper reports a production LLM knowledge-extraction pipeline that ingested 19 million facts from Wikipedia at 98.8% precision.","keywords":["knowledge graph construction","LLM-based knowledge extraction","ontology-guided prompting","fact grounding","corroboration and ranking","open-domain extraction","Wikipedia","production deployment"],"falsifier":"Take a random, stratified sample of the 19 million ingested triples, including long-tail entities and rare predicates, and have a fresh set of annotators independently re-check each triple against its cited provenance without seeing ODKE+'s confidence scores. If measured precision falls well below 98.8%, or if the Grounder's True/False decisions disagree with the re-check at a high rate, the central claim is not supported.","tokens_in":8367,"feed_emoji":"🕸️","tokens_out":9370,"duration_ms":84157,"temperature":0.7,"pith_summary":"ODKE+ is a production knowledge-graph pipeline that uses large language models to pull facts out of web pages and still keeps precision high enough to trust. The paper reports that the system ingested more than 19 million facts across 195 predicates from over 9 million Wikipedia pages at 98.8% audited precision, with grounding alone cutting hallucinated extractions by 35%. The core design is ontology-guided prompting: each entity type gets a compact schema snippet listing valid predicates, qualifiers, and units, so the LLM produces schema-consistent output without per-predicate hand-built rules. A second, lightweight LLM then checks each candidate fact against its source text, and a corroborator normalizes, aggregates, and ranks the survivors before ingestion. The result, if the numbers hold, turns LLM extraction from a research demonstration into a viable primary channel for keeping large ontologies fresh.","feed_headline":"19M facts at 98.8% precision: LLM pipeline feeds knowledge graph","feed_subtitle":"Ontology-guided prompts plus a second-LLM verifier cut hallucinations 35% and arrive 50 days faster.","key_machinery":"The central mechanism is ontology snippet generation: an offline pipeline inspects the KG, the ontology, and Wikidata, ranks predicates per entity type, enriches them with descriptions, domains, ranges, qualifiers, cardinalities, and normalization configs, then serializes this into a compact text fragment placed inside the LLM prompt. This keeps extractions aligned with schema constraints without manual per-predicate rules. The second load-bearing component is the Grounder, a separate LLM that receives the source context and an assertion of the extracted triple and answers True or False; only affirmative verdicts proceed to the Corroborator, which converts values into canonical units and for","core_discovery":"ODKE+ claims that ontology-guided prompting plus a lightweight grounding verifier makes LLM-based open-domain fact extraction accurate and fast enough to run continuously in a production knowledge graph. The pipeline detects which entities have missing or stale facts by watching page edits, retrieves the updated Wikipedia content, extracts candidate triples with either pattern rules or an LLM, asks a separate Grounder LLM whether each candidate is explicitly supported by the context, then normalizes, aggregates, and ranks survivors before ingestion. In the reported deployment, this produced 19 million accepted facts at 98.8% precision, reduced hallucinated extractions by 35% compared with un","pith_inferences":["Editorial inference: the 91% raw-LLM baseline and 35% hallucination reduction are aggregate numbers; reporting how grounding's benefit varies across the 195 predicates would show whether verifier effort should be concentrated on high-volume or high-risk predicate types.","Editorial inference: the reported 69-day head start of Wikipedia over Wikidata suggests a general freshness strategy—monitoring edit streams as a weak signal that knowledge changed—that could transfer to other public sources where the entity-to-source mapping is less deterministic.","Editorial inference: because ontology snippets are derived partly from existing KG usage patterns, the system is likely to reinforce the ontology's current coverage; testing whether ODKE+ can recover predicates or entity types absent from the snippet would clarify whether the approach is truly open-domain.","Editorial inference: the aggregate 98.8% precision may hide variation across long-tail entities and rare predicates, where corroboration has fewer independent sources; mapping that variation would make the production claims more transferable."],"forward_implications":["Knowledge graphs can treat LLM extraction as a first-class ingestion source rather than a research prototype, with new predicates added declaratively by updating the ontology snippet instead of writing extractor rules.","Freshness no longer requires full-corpus reprocessing: a streaming mode keyed to page-edit signals can keep high-priority entities current with under-two-hour latency.","A separate lightweight verifier after a generative extractor is a reusable design pattern for reducing hallucinations in structured extraction tasks beyond knowledge graphs.","The reported 35% hallucination reduction and 91% to 98.8% precision lift imply that most LLM errors surviving to the ranking stage are caught by grounding and corroboration, so downstream applications can rely on the ingested triples.","Focusing extraction on frequently stale predicates such as height, population, age, and net worth lets the system spend compute where facts change fastest."],"supporting_citations":[{"why":"Defines the Saga platform for continuous knowledge-graph construction and serving that ODKE+ builds on and extends.","marker":"Ilyas et al., 2022"},{"why":"Introduces the original ODKE extraction component whose architecture and extraction capabilities ODKE+ inherits.","marker":"Ilyas et al., 2023"},{"why":"Presents ODKE v2, the prior version that added multilingual support, streaming ingestion, and limited LLM-based extractors.","marker":"Qian et al., 2023"},{"why":"Provides the search-based knowledge-base completion approach that motivates ODKE+'s evidence retrieval and freshness design.","marker":"West et al., 2014"},{"why":"Supplies the normalization library used to convert surface forms like heights and dates into canonical values.","marker":"Duckling, 2016"},{"why":"Supplies the AutoML package used for learned scoring and ranking of candidate facts in the Corroborator.","marker":"LeDell and Poirier, 2020"}],"fun_headline_variants":["Ontology-guided LLM extraction: 19M facts, 98.8% precision","LLM pipeline cuts knowledge graph lag by 50 days","Second-LLM verifier slashes hallucinations 35%","ODKE+: Production-grade KG ingestion from web at scale","From Wikipedia to KG: 19M facts with ontology-guardrails"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The load-bearing premise is that the Grounder LLM's True/False verdicts are accurate and that the weekly audit of about 2,000 randomly sampled triples is representative; the paper does not report inter-annotator agreement, the audit sampling protocol, or independent validation of the Grounder's judgments, so all headline precision numbers inherit those assumptions.","fun_headline_variants_meta":{"raw":{"variants":["Ontology-guided LLM extraction: 19M facts, 98.8% precision","LLM pipeline cuts knowledge graph lag by 50 days","Second-LLM verifier slashes hallucinations 35%","ODKE+: Production-grade KG ingestion from web at scale","From Wikipedia to KG: 19M facts with ontology-guardrails"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000682,"raw_usage":{"total_tokens":2963,"prompt_tokens":801,"completion_tokens":2162,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":545,"completion_tokens_details":{"reasoning_tokens":2070}},"tokens_in":545,"tokens_out":2162,"duration_ms":14814,"temperature":1.0,"reasoning_tokens":2070,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T05:56:10.226373+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random, stratified sample of the 19 million ingested triples, including long-tail entities and rare predicates, and have a fresh set of annotators independently re-check each triple against its cited provenance without seeing ODKE+'s confidence scores. If measured precision falls well below 98.8%, or if the Grounder's True/False decisions disagree with the re-check at a high rate, the central claim is not supported.","supporting_citations":[{"cited_title":"Open Domain Knowledge Extraction for Knowledge Graphs","cited_arxiv_id":"2312.09424","evidence_quote":"Introduces the original ODKE extraction component whose architecture and extraction capabilities ODKE+ inherits."},{"cited_title":"Open Domain Knowledge Extraction for Knowledge Graphs","cited_arxiv_id":"2312.09424","evidence_quote":"Presents ODKE v2, the prior version that added multilingual support, streaming ingestion, and limited LLM-based extractors."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the normalization library used to convert surface forms like heights and dates into canonical values."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the AutoML package used for learned scoring and ranking of candidate facts in the Corroborator."}],"review_version":1}