{"id":"046007f7-2da0-46bc-bd0d-7a66376ef5e9","arxiv_id":"2412.20005","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A multi-agent LLM system with schema configuration and a case repository improves knowledge extraction from web pages, PDFs, and other raw text across domains.","lead":"OneKE is a ready-to-run software package that uses several AI agents working together to pull structured facts out of messy sources like web pages and PDF books. It is useful for anyone building knowledge graphs or retrieval systems who wants to avoid hand-writing extraction code for each new document type.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported F1 gains hinge on the Case Repository being built exclusively from non-test data, but the paper never specifies how the repository was seeded for CrossNER and NYT-11-HRL; automatic updating after each task (Sec. 2.4) makes test leakage plausible.","rationale":"The reader's weakest_assumption points directly at the repository construction and its separation from test data; my reading of the paper reaches the same conclusion. The improvement mechanism of OneKE is entirely mediated by retrieval from the Case Repository, so contamination there would invalidate the empirical comparisons in Figure 2. The paper gives no details on repository seeding, the split protocol, or the exact evaluation setup, and the automatic-update statement in Section 2.4 raises a concrete leakage channel. This does not require accusing the authors of misconduct; it is an unidentified methodological condition that must be stated and checked before the central claim can be accepted. The paper does have real supporting artifacts—open-source code and video demonstrations—which allow the test to be run. Since the concern can be resolved by inspecting and re-running the released system, retaining a CONDITIONAL verdict is appropriate; no change to the reader's judgment is warranted.","tokens_in":5942,"tokens_out":2933,"duration_ms":29431,"concrete_test":"Inspect the released code (github.com/zjunlp/OneKE) for the experiment scripts that produced Figure 2. Check whether the Case Repository is seeded exclusively from the training splits of CrossNER and NYT-11-HRL, and whether any evaluation of the test set triggers the automatic repository update described in Section 2.4. If repository contents include test-set-derived cases, re-run the ablation with a repository built only from training data; if the F1 gains over Vanilla vanish, the paper's central efficacy claim is unsupported by the current evidence.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that case retrieval and reflection improve extraction—requires that the retrieved correct/bad cases come from a repository whose contents are independent of the test set. Section 2.4 states the Case Repository 'will be automatically updated once a knowledge extraction task is completed' and that correct answers are stored with reasoning steps; it never states what data populated the repository during the benchmark evaluations. If any test-set example (or a near-duplicate) was inserted before or during evaluation, the Extraction Agent's retrieval (all-MiniLM-L6-v2 + FuzzyWuzzy, Top-2) would retrieve the answer itself, trivially raising F1. The paper reports only bar-chart F1s without exact numbers, error bars, or a description of how Vanilla differs from the full pipeline, so the magnitude of the claimed improvement cannot be checked. This is the weakest load-bearing assumption in the empirical support.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces OneKE, a dockerized, schema-guided knowledge extraction system built around three LLM agents (Schema Agent, Extraction Agent, Reflection Agent) and a configure knowledge base that stores schemas and historical extraction cases. The system is designed to handle raw web HTML, PDF book chapters, and other document types, and to support user-defined or LLM-deduced schemas. The authors report experiments on two benchmark datasets (CrossNER for NER and NYT-11-HRL for RE), showing that case retrieval and case reflection improve F1 over a vanilla prompting baseline, and they present two case studies on web news extraction and book knowledge extraction. The code and a demo video are open-sourced.","tokens_in":6161,"tokens_out":3609,"duration_ms":40548,"significance":"If the reported gains hold, OneKE would be a useful engineering contribution: it packages a multi-agent LLM pipeline with retrieval-based few-shot case adaptation, reflection-based error correction, and configurable schemas into a dockerized system that can be applied to real-world heterogeneous inputs. The open-sourced implementation, the support for multiple LLM backends, and the explicit emphasis on error debugging via a Case Repository are strengths that make the system potentially valuable to practitioners. However, the current evidence for efficacy is too thin to establish the central claim: the quantitative results are presented only as unlabeled bar charts, with no external baselines, no error bars, and no specification of the repository construction protocol. The paper is therefore best treated as a promising system description whose empirical evaluation needs substantial strengthening before the claimed improvements can be taken as demonstrated.","major_comments":[{"comment":"The central performance claim—that case retrieval and case reflection improve extraction—is supported only by two bar charts with no numeric F1 values, no error bars, no number of runs, and no external baselines. The y-axis ranges and the magnitude of the improvements cannot be verified from the figure, and the text does not report the standard deviation across evaluation folds or random seeds. Please report exact F1 numbers (ideally in a table), the evaluation protocol, and the number of repetitions, and compare against at least one non-ablation baseline (e.g., a standard few-shot LLM prompt or an existing IE system such as InstructUIE or AgentRE). Without these, the claimed efficacy is not quantitatively established.","section":"§3, Figure 2"},{"comment":"The Case Repository construction is not specified, and this is a load-bearing data-hygiene issue. The paper states that the repository 'will be automatically updated once a knowledge extraction task is completed' and stores correct answers with reasoning steps, but it never says what data seeded the repository during the CrossNER and NYT-11-HRL evaluations. If any test-set example, or a near-duplicate, was inserted into the repository before or during evaluation, the retrieval step (all-MiniLM-L6-v2 plus FuzzyWuzzy, Top-2) could retrieve the ground-truth answer for the current input, inflating the reported F1 gains in a way that does not reflect genuine generalization. Please specify how the repository was populated for the benchmark experiments (e.g., using only training/development examples, with manual verification of labels and reasoning steps, and with deduplication against the test sets), and state clearly whether any test instance was excluded.","section":"§2.4"},{"comment":"Many implementation choices that directly affect the results are unreported: the retrieval top-k is said to default to two, but no search is reported over this value; the self-consistency mechanism is mentioned but its sample size, voting procedure, and uncertainty threshold are not given; and the prompt templates, chunking parameters, temperature, and model versions are not described. Because the paper's claims are about the pipeline's components rather than a new model architecture, these details are necessary for reproducibility and for attributing the gains to case retrieval versus prompt construction. Please provide the exact configuration (or a configuration file in the repository) and the prompts used in the experiments.","section":"§2.2–§2.3"}],"minor_comments":[{"comment":"The term 'configure knowledge base' appears in the abstract and Section 2.4; the intended wording is likely 'configured knowledge base' or 'configurable knowledge base'.","section":"Abstract and §2.4"},{"comment":"In the Book Knowledge Extraction example, the JSON output contains a missing closing quote in '\"name\": \"Harry Potter}' (should be \"Harry Potter\"), and the figure's font is very small; a larger, high-resolution version would improve readability.","section":"Figure 3"},{"comment":"CrossNER is described as a cross-domain NER dataset, but only an aggregate F1 is shown; reporting per-domain scores (e.g., politics, science, etc.) would strengthen the cross-domain claim.","section":"§3"},{"comment":"The paper says users can define new data types and add custom preprocessing methods, but no interface details are given; a pointer to the relevant code or documentation in the repository would help users adopt the system.","section":"§2.1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a short companion/system paper, and the open-sourced code is a genuine asset. I would be willing to accept after the authors make the evaluation transparent: exact numbers, variances, external baselines, and a clear statement of how the Case Repository was seeded for the benchmarks. If the repository-leakage concern cannot be resolved, the empirical claims should be substantially weakened; the paper could still be published as a system/demo if framed accordingly."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: OneKE is a genuinely useful integrated extraction system, but its own evaluation is too thin to support the efficacy claim — the case repository seeding needs clarification.\n\nWhat's actually new: The system's three-agent design (Schema, Extraction, Reflection) with a configurable, updatable knowledge base is a solid engineering contribution. The integration is real and the code is open-sourced. Following AgentRE for reflection and citing it is honest. The comparison of case retrieval and reflection on CrossNER and NYT-11-HRL is new in this specific form, even though each component has prior art.\n\nWhat works well: OneKE handles raw HTML/PDF, supports custom and self-deduced schemas, and plugs into different LLMs. The case repository's separation into correct and bad cases, with reasoning steps, is a sensible way to do few-shot and self-correction. The case studies on news and books show the intended usage.\n\nWhere it's soft: The evaluation is the weak link. Figure 2 is the entire evidence: two bar charts with no exact F1 values, no error bars, and no external baselines. The paper also never describes how Vanilla differs from the full pipeline. More critically, Section 2.4 says the Case Repository 'will be automatically updated once a knowledge extraction task is completed,' but never states what data seeded the repository during the benchmark runs. If any test example or near-duplicate was inserted before or during evaluation, the retrieval (all-MiniLM-L6-v2 + FuzzyWuzzy, top-2) would return the answer itself and inflate F1. This is a data-hygiene problem, not circular reasoning within the claims, but it's load-bearing. The reported gains from case retrieval and reflection cannot be trusted without a clear statement of repository provenance and separation from test data.\n\nAlso minor: the application case studies are illustrative, not evidential. The abstract's 'efficacy' language overpromises relative to what's shown.\n\nWho this is for: people building KG or RAG pipelines who want a working tool, not researchers looking for a new method. The system itself is likely to be useful to practitioners.\n\nRecommendation: I'd engage with it, but only after the authors provide numeric results, baselines, and a clear account of how the Case Repository was built for each benchmark. If the repository provenance checks out, the paper is a reasonable demo/systems contribution; if not, the central claim collapses. A serious referee should see this at review time.","headline":"OneKE is a well-engineered integration of known components, but the paper's two-bar-chart evidence and unspecified case-repository seeding make the efficacy claim unverifiable.","tokens_in":6669,"tokens_out":3039,"would_cite":false,"duration_ms":29659,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"OneKE claims that a schema-guided three-agent system with a case repository extracts knowledge from raw web pages and PDFs better than plain LLM prompting.","keywords":["knowledge extraction","large language models","multi-agent system","schema-guided extraction","case repository","reflection mechanism","dockerized system","relation extraction"],"falsifier":"Re-run the two benchmark evaluations with an empty Case Repository, or with the repository seeded only from the training portions of the same datasets, and compare F1 to the advertised numbers; if the advantage disappears, the earlier gains depended on leakage from test examples into the repository.","tokens_in":5798,"feed_emoji":"🤖","tokens_out":4975,"duration_ms":41276,"temperature":0.7,"pith_summary":"OneKE is a dockerized knowledge-extraction system designed to pull structured knowledge out of raw, messy inputs—web HTML pages and PDF book chapters—rather than clean plain text only. The paper claims that a schema-guided pipeline of three agents (Schema, Extraction, Reflection) plus a knowledge base of past extraction cases outperforms vanilla LLM prompting on named-entity recognition and relation extraction, with the largest gains coming from retrieving successful cases as few-shot demonstrations. If the claim holds, OneKE gives non-specialists a way to configure extraction schemas in plain text and get structured JSON back, without any fine-tuning.","feed_headline":"Agent trio beats plain LLM prompting on knowledge extraction","feed_subtitle":"Schema-guided agents with an error-case memory pull structured JSON from raw HTML and PDF text, no fine-tuning required.","key_machinery":"The mechanism is a three-agent loop supported by a configure knowledge base. The Schema Agent converts a user instruction—with or without an explicit schema—into a unified JSON output schema, using a Schema Repository of predefined Pydantic-encoded templates; the Extraction Agent retrieves similar Correct Cases from the Case Repository via semantic similarity plus string matching and appends them as few-shot examples before calling an LLM; and the Reflection Agent retrieves relevant Bad Cases with their reflective analyses to prompt the LLM to revise its initial output. The Case Repository is automatically updated after each completed task, so the system's memory of successful reasoning steps and past mistakes is what carries the claimed improvement.","core_discovery":"The core discovery is that a system integrating schema generation, case-based retrieval, and reflection-based error correction yields better extraction performance than directly prompting the same LLM. On the CrossNER named-entity-recognition benchmark and the NYT-11-HRL relation-extraction benchmark, the paper reports F1 improvements from both Case Retrieval and Case Reflection, and observes that Case Retrieval contributes the most, particularly for the more complex relation-extraction task where intermediate reasoning steps from correct cases matter. OneKE is claimed to generalize across domains and data formats by treating schema selection and self-schema deduction as first-class steps.","pith_inferences":["The magnitude of the reported gains may not transfer to domains where the Case Repository is sparse; retrieval quality depends on the semantic similarity matcher finding genuinely comparable examples.","An automatic Case Repository that absorbs every completed task risks accumulating noisy or contradictory cases, so in production the repository likely needs curation or scoring rather than blind self-updating.","A natural, testable extension is measuring how the F1 gain scales with repository size and retrieval top-k, which the paper does not report.","If reflection depends on retrieving truly relevant bad cases, then any error in the retrieval step could make the Reflection Agent reinforce, rather than fix, a mistake."],"forward_implications":["A user can feed a raw HTML article or a PDF chapter into OneKE, choose or describe a schema in plain text, and receive structured JSON without any model training.","The Case Retrieval component should keep improving extraction as more correct cases accumulate, particularly for relation extraction where reasoning chains matter.","Because the underlying LLM is swappable, OneKE can adopt newer open-source or API models without changing the extraction pipeline.","The Dockerized packaging lets the system drop into knowledge-graph construction and retrieval-augmented generation workflows as a reusable extraction layer."],"supporting_citations":[{"why":"Supplies the multi-agent reflection design that OneKE follows for its Reflection Agent.","marker":"[6]"},{"why":"Prior LLM-based incremental knowledge graph construction system that OneKE positions against as a schema-guided alternative.","marker":"[4]"},{"why":"Prior unified instruction-tuned information extraction approach that OneKE extends to an agent-based setting.","marker":"[8]"},{"why":"Survey of LLM-based generative information extraction that provides the broader context OneKE claims to advance.","marker":"[10]"},{"why":"Shows structured information extraction from scientific text with LLMs, motivating OneKE's domain-application use cases.","marker":"[2]"}],"fun_headline_variants":["Schema-guided agents with case retrieval beat direct LLM prompting","Case retrieval, not reflection, drives LLM extraction gains","Multi-agent schema system extracts knowledge from Web and PDF","Dockerized LLM agents with error memory improve extraction"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported F1 gains presuppose that the Case Repository used in the CrossNER and NYT-11-HRL evaluations was seeded from data that does not overlap with the test sets, and that those stored cases are correctly labeled.","fun_headline_variants_meta":{"raw":{"variants":["Schema-guided agents with case retrieval beat direct LLM prompting","Case retrieval, not reflection, drives LLM extraction gains","Multi-agent schema system extracts knowledge from Web and PDF","Dockerized LLM agents with error memory improve extraction"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00095,"raw_usage":{"total_tokens":3976,"prompt_tokens":788,"completion_tokens":3188,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":404,"completion_tokens_details":{"reasoning_tokens":3122}},"tokens_in":404,"tokens_out":3188,"duration_ms":22465,"temperature":1.0,"reasoning_tokens":3122,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T23:40:19.672389+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the two benchmark evaluations with an empty Case Repository, or with the repository seeded only from the training portions of the same datasets, and compare F1 to the advertised numbers; if the advantage disappears, the earlier gains depended on leakage from test examples into the repository.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the multi-agent reflection design that OneKE follows for its Reflection Agent."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Survey of LLM-based generative information extraction that provides the broader context OneKE claims to advance."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Shows structured information extraction from scientific text with LLMs, motivating OneKE's domain-application use cases."}],"review_version":1}