REVIEW 3 major objections 4 minor 1 cited by
OneKE: A Dockerized Schema-Guided LLM Agent-based Knowledge Extraction System
T0 review · 3 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read OneKE claims that a schema-guided three-agent system with a case repository extracts knowledge from raw web pages and PDFs better than plain LLM prompting.
desk verdict OneKE is a well-engineered integration of known components, but the paper's two-bar-chart evidence and unspecified case-repository seeding make the efficacy claim unverifiable. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The mechanism is a three-agent loop supported by a configure knowledge base. The Schema Agent converts a user instruction—with or without an explicit schema—into a unified JSON output schema, using a Schema Repository of predefined Pydantic-encoded templates; the Extraction Agent retrieves similar Correct Cases from the Case Repository via semantic similarity plus string matching and appends them as few-shot examples before calling an LLM; and the Reflection Agent retrieves relevant Bad Cases with their reflective analyses to prompt the LLM to revise its initial output. The Case Repository is automatically updated after each completed task, so the system's memory of successful reasoning steps and past mistakes is what carries the claimed improvement.
What would settle it
Re-run the two benchmark evaluations with an empty Case Repository, or with the repository seeded only from the training portions of the same datasets, and compare F1 to the advertised numbers; if the advantage disappears, the earlier gains depended on leakage from test examples into the repository.
Extended reading notes
Core claim
The core discovery is that a system integrating schema generation, case-based retrieval, and reflection-based error correction yields better extraction performance than directly prompting the same LLM. On the CrossNER named-entity-recognition benchmark and the NYT-11-HRL relation-extraction benchmark, the paper reports F1 improvements from both Case Retrieval and Case Reflection, and observes that Case Retrieval contributes the most, particularly for the more complex relation-extraction task where intermediate reasoning steps from correct cases matter. OneKE is claimed to generalize across domains and data formats by treating schema selection and self-schema deduction as first-class steps.
Load-bearing premise
The reported F1 gains presuppose that the Case Repository used in the CrossNER and NYT-11-HRL evaluations was seeded from data that does not overlap with the test sets, and that those stored cases are correctly labeled.
Editorial extensions
If this is right
- A user can feed a raw HTML article or a PDF chapter into OneKE, choose or describe a schema in plain text, and receive structured JSON without any model training.
- The Case Retrieval component should keep improving extraction as more correct cases accumulate, particularly for relation extraction where reasoning chains matter.
- Because the underlying LLM is swappable, OneKE can adopt newer open-source or API models without changing the extraction pipeline.
- The Dockerized packaging lets the system drop into knowledge-graph construction and retrieval-augmented generation workflows as a reusable extraction layer.
Reading between the lines
- The magnitude of the reported gains may not transfer to domains where the Case Repository is sparse; retrieval quality depends on the semantic similarity matcher finding genuinely comparable examples.
- An automatic Case Repository that absorbs every completed task risks accumulating noisy or contradictory cases, so in production the repository likely needs curation or scoring rather than blind self-updating.
- A natural, testable extension is measuring how the F1 gain scales with repository size and retrieval top-k, which the paper does not report.
- If reflection depends on retrieving truly relevant bad cases, then any error in the retrieval step could make the Reflection Agent reinforce, rather than fix, a mistake.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces OneKE, a dockerized, schema-guided knowledge extraction system built around three LLM agents (Schema Agent, Extraction Agent, Reflection Agent) and a configure knowledge base that stores schemas and historical extraction cases. The system is designed to handle raw web HTML, PDF book chapters, and other document types, and to support user-defined or LLM-deduced schemas. The authors report experiments on two benchmark datasets (CrossNER for NER and NYT-11-HRL for RE), showing that case retrieval and case reflection improve F1 over a vanilla prompting baseline, and they present two case studies on web news extraction and book knowledge extraction. The code and a demo video are open-sourced.
Significance. If the reported gains hold, OneKE would be a useful engineering contribution: it packages a multi-agent LLM pipeline with retrieval-based few-shot case adaptation, reflection-based error correction, and configurable schemas into a dockerized system that can be applied to real-world heterogeneous inputs. The open-sourced implementation, the support for multiple LLM backends, and the explicit emphasis on error debugging via a Case Repository are strengths that make the system potentially valuable to practitioners. However, the current evidence for efficacy is too thin to establish the central claim: the quantitative results are presented only as unlabeled bar charts, with no external baselines, no error bars, and no specification of the repository construction protocol. The paper is therefore best treated as a promising system description whose empirical evaluation needs substantial strengthening before the claimed improvements can be taken as demonstrated.
major comments (3)
- [§3, Figure 2] The central performance claim—that case retrieval and case reflection improve extraction—is supported only by two bar charts with no numeric F1 values, no error bars, no number of runs, and no external baselines. The y-axis ranges and the magnitude of the improvements cannot be verified from the figure, and the text does not report the standard deviation across evaluation folds or random seeds. Please report exact F1 numbers (ideally in a table), the evaluation protocol, and the number of repetitions, and compare against at least one non-ablation baseline (e.g., a standard few-shot LLM prompt or an existing IE system such as InstructUIE or AgentRE). Without these, the claimed efficacy is not quantitatively established.
- [§2.4] The Case Repository construction is not specified, and this is a load-bearing data-hygiene issue. The paper states that the repository 'will be automatically updated once a knowledge extraction task is completed' and stores correct answers with reasoning steps, but it never says what data seeded the repository during the CrossNER and NYT-11-HRL evaluations. If any test-set example, or a near-duplicate, was inserted into the repository before or during evaluation, the retrieval step (all-MiniLM-L6-v2 plus FuzzyWuzzy, Top-2) could retrieve the ground-truth answer for the current input, inflating the reported F1 gains in a way that does not reflect genuine generalization. Please specify how the repository was populated for the benchmark experiments (e.g., using only training/development examples, with manual verification of labels and reasoning steps, and with deduplication against the test sets), and state clearly whether any test instance was excluded.
- [§2.2–§2.3] Many implementation choices that directly affect the results are unreported: the retrieval top-k is said to default to two, but no search is reported over this value; the self-consistency mechanism is mentioned but its sample size, voting procedure, and uncertainty threshold are not given; and the prompt templates, chunking parameters, temperature, and model versions are not described. Because the paper's claims are about the pipeline's components rather than a new model architecture, these details are necessary for reproducibility and for attributing the gains to case retrieval versus prompt construction. Please provide the exact configuration (or a configuration file in the repository) and the prompts used in the experiments.
minor comments (4)
- [Abstract and §2.4] The term 'configure knowledge base' appears in the abstract and Section 2.4; the intended wording is likely 'configured knowledge base' or 'configurable knowledge base'.
- [Figure 3] In the Book Knowledge Extraction example, the JSON output contains a missing closing quote in '"name": "Harry Potter}' (should be "Harry Potter"), and the figure's font is very small; a larger, high-resolution version would improve readability.
- [§3] CrossNER is described as a cross-domain NER dataset, but only an aggregate F1 is shown; reporting per-domain scores (e.g., politics, science, etc.) would strengthen the cross-domain claim.
- [§2.1] The paper says users can define new data types and add custom preprocessing methods, but no interface details are given; a pointer to the relevant code or documentation in the repository would help users adopt the system.
Circularity Check
No significant circularity: OneKE's claims are system-design and empirical, with no derivation that reduces to its inputs.
full rationale
OneKE is a system paper describing a multi-agent knowledge extraction pipeline and reporting benchmark F1 comparisons. There is no mathematical derivation or fitted-parameter chain whose output is equivalent to an input by construction. The Reflection Agent design is attributed to AgentRE [6], but that citation is a design provenance statement, not the evidential basis for the reported F1 gains; the evaluations compare Vanilla, Case Retrieval, and Case Reflection conditions directly on CrossNER and NYT-11-HRL. The Case Repository's automatic updating after completed tasks (Sec. 2.4) raises a possible data-hygiene concern about whether test examples could enter the retrieval pool, but that is an empirical soundness issue, not circular reasoning: the paper never claims the repository is constructed from the test labels, and no equation or definition forces the reported improvements. The absence of exact numbers, error bars, and a precise description of the Vanilla baseline also limits verifiability but does not constitute circularity. Accordingly, no specific circular step can be quoted and exhibited, and the appropriate score is 0.
Assumptions & free parameters
free parameters (2)
- retrieval_top_k =
2
- self_consistency_sample_size
assumptions (4)
- domain assumption LLMs can follow structured output schemas and perform self-reflective correction.
- domain assumption CrossNER and NYT-11-HRL are valid benchmarks with reliable labels.
- domain assumption Langchain document_loaders parse HTML, PDF, and Word faithfully.
- ad hoc to paper Case Repository cases are representative and correctly labeled.
Cite this review
Pith. "Pith review of OneKE: A Dockerized Schema-Guided LLM Agent-based Knowledge Extraction System." pith.science (2026). https://pith.science/paper/U7FMJLBY
@misc{pith2026241220005,
author = {Pith},
title = {Pith review of: OneKE: A Dockerized Schema-Guided LLM Agent-based Knowledge Extraction System},
year = {2026},
howpublished = {\url{https://pith.science/paper/U7FMJLBY}},
note = {Machine review of arXiv:2412.20005}
}
read the original abstract
We introduce OneKE, a dockerized schema-guided knowledge extraction system, which can extract knowledge from the Web and raw PDF Books, and support various domains (science, news, etc.). Specifically, we design OneKE with multiple agents and a configure knowledge base. Different agents perform their respective roles, enabling support for various extraction scenarios. The configure knowledge base facilitates schema configuration, error case debugging and correction, further improving the performance. Empirical evaluations on benchmark datasets demonstrate OneKE's efficacy, while case studies further elucidate its adaptability to diverse tasks across multiple domains, highlighting its potential for broad applications. We have open-sourced the Code at https://github.com/zjunlp/OneKE and released a Video at http://oneke.openkg.cn/demo.mp4.
Figures
Forward citations
Cited by 1 Pith paper
-
ODKE+: Ontology-Guided Open-Domain Knowledge Extraction with LLMs
A production system using ontology-guided prompts and two LLM stages extracted 19 million high-confidence facts from Wikipedia with 98.8% reported precision.
Reference graph
Works this paper leans on
-
[1]
Xiang Chen, Ningyu Zhang, Xin Xie, Shumin Deng, Yunzhi Yao, Chuanqi Tan, Fei Huang, Luo Si, and Huajun Chen. 2022. Knowprompt: Knowledge-aware prompt-tuning with synergistic optimization for relation extraction. In WWW
work page 2022
-
[2]
John Dagdelen, Alexander Dunn, Sanghoon Lee, Nicholas Walker, Andrew S Rosen, Gerbrand Ceder, Kristin A Persson, and Anubhav Jain. 2024. Structured information extraction from scientific text with large language models. Nature Communications 15, 1 (2024), 1418
work page 2024
-
[3]
Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, and Haofen Wang. 2023. Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997 (2023)
arXiv 2023
-
[4]
Yassir Lairgi, Ludovic Moncla, Rémy Cazabet, Khalid Benabdeslem, and Pierre Cléau. 2024. iText2KG: Incremental Knowledge Graphs Construction Using Large Language Models. CoRR abs/2409.03284 (2024). arXiv:2409.03284
arXiv 2024
-
[5]
Bo Li, Gexiang Fang, Yang Yang, Quansen Wang, Wei Ye, Wen Zhao, and Shikun Zhang. 2023. Evaluating ChatGPT’s Information Extraction Capabilities: An Assessment of Performance, Explainability, Calibration, and Faithfulness. arXiv preprint arXiv:2304.11633 (2023)
arXiv 2023
-
[6]
Yuchen Shi, Guochao Jiang, Tian Qiu, and Deqing Yang. 2024. AgentRE: An Agent-Based Framework for Navigating Complex Information Landscapes in Relation Extraction. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management, CIKM 2024 . ACM
work page 2024
-
[7]
Nan Sun, Ming Ding, Jiaojiao Jiang, Weikang Xu, Xiaoxing Mo, Yonghang Tai, and Jun Zhang. 2023. Cyber threat intelligence mining for proactive cybersecurity defense: a survey and new perspectives.IEEE Communications Surveys & Tutorials 25, 3 (2023), 1748–1774
work page 2023
-
[8]
Xiao Wang, Weikang Zhou, Can Zu, et al. 2023. InstructUIE: Multi-task Instruc- tion Tuning for Unified Information Extraction. CoRR abs/2304.08085 (2023). arXiv:2304.08085
arXiv 2023
Show all 10 references
-
[9]
Xiang Wei, Xingyu Cui, Ning Cheng, Xiaobin Wang, Xin Zhang, Shen Huang, Pengjun Xie, Jinan Xu, Yufeng Chen, Meishan Zhang, et al . 2024. Chatie: Zero-shot information extraction via chatting with chatgpt. arXiv preprint arXiv:2302.10205 (2024)
2024 arXiv
-
[10]
Derong Xu, Wei Chen, Wenjun Peng, Chao Zhang, and Tong et al. Xu. 2024. Large language models for generative information extraction: A survey. Frontiers of Computer Science (2024)
2024
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.