{"id":"d30d95a0-cf4a-43d2-a7f6-2ee0620ba05d","arxiv_id":"2506.06881","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"KnowCoder-V2 augments deep research with offline knowledge organization and code-based knowledge computation, reporting gains on information extraction, KBQA, and LLM-judged report generation.","lead":"KnowCoder-V2 is a research agent that first turns large document collections into structured Python knowledge objects, then writes and runs analysis code over those objects when answering questions. The paper reports that this design, wrapped in an offline-plus-online deep research framework, beats commercial deep research tools on a self-scored report quality test.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The report-generation headline rests on an unblinded, self-constructed evaluation: DeepSeek-R1 judges reports while also powering parts of KDR; the 8.9 vs 7.7 advantage is not independently established.","rationale":"The reader's weakest assumption correctly isolates the report-generation evaluation as the load-bearing point. I agree with that identification and add the precise internal-conflict detail: Figure 3's caption states that DeepSeek-R1 handles the non-knowledge-analysis parts of KDR, while Section 5.3.2 says scores are averaged from DeepSeek-R1 and DeepSeek-V3. One of the two judges therefore belongs to the system under evaluation, and the evaluation dataset is neither released nor independently vetted. This makes the headline claim of superior report quality unverified rather than merely unvalidated. The rest of the paper does show genuine engineering value: the IE experiments use standard external benchmarks, the KBQA result on WebQSP is against established baselines, and the code-generation formulation is coherent with the KnowCoder line of work. Those results support a conditional acceptance. The report-generation claim specifically should be treated as conditional on external validation, which matches the reader's verdict. I do not see a reason to move to REJECT, because the concern is about evidence quality rather than internal inconsistency in the extraction and KBQA results.","tokens_in":24222,"tokens_out":3131,"duration_ms":34556,"concrete_test":"Run a blinded human evaluation: have at least three expert annotators score the reports, or a released representative subset of the self-constructed scientist-achievement dataset, on completeness, thoroughness, factuality, coherence, and insight, without knowing which system produced each report, and report inter-annotator agreement. If the human-averaged scores do not reproduce a KnowCoder-V2 advantage over OpenAI Deep Research, ideally with paired significance testing, the headline report-generation claim is unsupported and should be downgraded. As a complementary check, re-run the same comparison with a judge outside the DeepSeek family (e.g., GPT-4o or human) to quantify the judge-confound magnitude.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (Abstract; Section 5.3.2, Table 6) is that KDR+KnowCoder-V2 produces higher-quality deep-research reports than OpenAI Deep Research (8.9 vs 7.7 average) and Grok3 (6.8). For this claim to hold, the report scores must measure report quality and insight fairly. That condition is the least secure point in the paper. Three facts jointly undermine it: (1) the dataset is self-constructed and unreleased (Section 5.1); (2) scores are averaged from DeepSeek-V3 and DeepSeek-R1, with no human evaluation or secondary judge (Section 5.3.2, Figure 1); and (3) DeepSeek-R1 explicitly \"handle[s] the parts of the KDR framework that are not related to knowledge analysis tasks\" (Figure 3 caption), so one of the two judges is part of the system being evaluated. This is not merely an independence nicety: if the judge has a systematic preference for KDR outputs, or if \"insight\" is proxied by the code-computation artifacts that KDR is designed to produce, the margin may reflect the judge rather than actual report quality. The paper also does not report ablations or error analysis separating the contribution of KnowCoder-V2, the offline knowledge base, and the human-in-the-loop ontology design to the report-generation score. The IE and KBQA results are on standard benchmarks and can stand on their own, but they do not establish the stronger deep-research claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes KnowCoder-V2, a 14B code-oriented LLM (fine-tuned from Qwen-2.5-Coder-14B-Instruct with LoRA) that represents ontologies as Python classes and knowledge instances as Python objects, and uses unified code generation both for knowledge organization (instantiation code) and knowledge reasoning (analysis code). The authors also introduce the Knowledgeable Deep Research (KDR) framework, which adds an offline knowledge organization phase and an online knowledge computation cycle to conventional deep research systems. Evaluation covers taxonomy expansion, English and Chinese information extraction (NER, RE, ED, EAE), a robustness benchmark (RUIE), KBQA on WebQSP, and a report-generation task on a self-constructed scientist-achievement dataset. The paper reports strong gains over prior IE models on most benchmarks, state-of-the-art Hits@1 on WebQSP, and higher LLM-judged report quality than OpenAI Deep Research, Grok3, Qwen3 Deep Research, and WebThinker.","tokens_in":24475,"tokens_out":3899,"duration_ms":44596,"significance":"If the headline claim is correct, the paper would show that a compact 14B model, combined with an offline knowledge base and code-based computation, can outperform large commercial deep research agents on report quality and insight, while also advancing the state of the art in universal information extraction. The paper's strengths include a broad evaluation across many standard IE benchmarks, a new robustness benchmark comparison, and a clean conceptual unification of organization and reasoning through code generation, with no fitted constants or circular derivations in the IE and KBQA experiments. The IE and KBQA results are independently meaningful and would likely stand on their own. However, the distinguishing claim made in the abstract and in Section 5.3.2 — that the KDR framework 'can generate high-quality reports with insightful analytical results' — is currently supported only by a self-constructed, unreleased dataset scored by two LLM judges, one of which is part of the evaluated system. The significance of the paper as a whole therefore depends on whether this evaluation weakness can be repaired; the underlying idea and the IE contributions are solid.","major_comments":[{"comment":"The report-generation claim, which is the paper's headline, rests on an evaluation that is not independent of the system. Scores are averaged from DeepSeek-V3 and DeepSeek-R1, and the Figure 3 caption explicitly states that DeepSeek-R1 'handle[s] the parts of the KDR framework that are not related to knowledge analysis tasks.' Thus one of the two judges is also part of the KDR pipeline being evaluated. The dataset is self-constructed and unreleased (Section 5.1), no human evaluation is reported, and no inter-judge agreement is given. This is a load-bearing weakness: the 8.9 versus 7.7 average advantage over OpenAI Deep Research (Table 6) may reflect a systematic judge preference for outputs that resemble KDR's own computational artifacts rather than a generalizable quality advantage. Please add a human evaluation on a sample of reports, use at least one judge that is not involved in the KDR pipeline, report agreement statistics, and release the dataset and generated reports. If these are not feasible, the deep-research claim should be substantially softened.","section":"§5.3.2, Table 6; Figure 3 caption"},{"comment":"The paper states in Section 4.3 that for ontology alignment and knowledge reasoning tasks, 'we are currently generating the corresponding training data and plan to release the trained models in the future.' This means KnowCoder-V2 is not trained on report-generation or KBQA-style reasoning tasks in this paper. Yet the report-generation experiments in Table 6 attribute the improvement to KnowCoder-V2's knowledge computation capability. This is a mismatch between the method description and the evaluation: the reported gains could come from the KDR framework, the base model's general code ability, or the human-in-the-loop ontology design, rather than from the claimed fine-tuned capabilities of KnowCoder-V2. The authors should clarify exactly which components of KnowCoder-V2 are used in the report-generation and KBQA experiments, and ideally provide a version of the model that is actually trained on the reasoning tasks, or redesign the claims to match the evaluated system.","section":"§4.3, §5.3.2"},{"comment":"There is no ablation isolating the contribution of the knowledge computation cycle to report quality. The comparison in Table 6 is between the full KDR+KnowCoder-V2 system and closed- or open-source deep research systems, but it does not compare against KDR without the knowledge computation cycle, KDR with a base Qwen-2.5-Coder model, or KDR with a different LLM in place of KnowCoder-V2. As a result, the statement that 'Due to the knowledge computation steps, KnowCoder-V2 is able to generate more comprehensive and insightful experimental results' (Section 5.3.2) is not directly supported by the experiments. Such ablations are needed to attribute the reported 8.9 score to the proposed components.","section":"§5.3.2, Table 6"},{"comment":"The KBQA evaluation is limited to a single dataset, WebQSP, and the experimental setup uses the ground-truth topic entity as initial context, following ToG. The paper reports Hits@1 but no variance or significance testing, and no analysis of failure cases. Given that this is one of only two knowledge-reasoning evaluations and that the model is not trained on reasoning data (Section 4.3), a single dataset is too thin to support the claim that 'KnowCoder-V2 is capable of generating accurate analysis code for the KBQA task.' Additional KBQA benchmarks or a more careful statement of scope would strengthen this section.","section":"§5.3.2, Table 5"},{"comment":"Table 1 groups KnowCoder-V2 under the '1-shot Setting' alongside NL and CodeTaxo, but the paper never specifies what the one-shot exemplar is, how KnowCoder-V2 is prompted for taxonomy expansion, or how the two-stage alignment with SimCSE and LLM re-ranking (Section 4.1.2) is combined with the one-shot protocol. This makes the strong results on WordNet and Graphine difficult to reproduce and difficult to interpret. Please provide the exact prompt and the one-shot example, or clarify whether KnowCoder-V2 in Table 1 is used as a prompting method or as a fine-tuned model.","section":"§5.1, Table 1"}],"minor_comments":[{"comment":"The sentence 'Table ?? presents the statistics of these taxonomies' contains an unresolved table reference; the table and its statistics are missing from the manuscript.","section":"§5.1"},{"comment":"There are numerous typos and grammatical errors that should be corrected, including 'Insigt' for 'Insight' in Section 5.3.2, 'exising' in the Introduction, 'eveluate' in Section 5.1, 'trianed' in Section 4.1.2, and 'filed search_results' in Section 4.2.2.","section":"Throughout"},{"comment":"The table headers are inconsistent: some columns are labeled 'Avg' while others are left blank or contain dashes, and KnowCoder-V1's Chinese NER entries are listed as 40.6 and 16.3 with no overall average. Please unify the notation and explain all dashes.","section":"Tables 2 and 3"},{"comment":"The paper reports average improvements of 3.03% and 2.92% F1 over 'all baselines' but does not report standard deviations or significance tests. Given that many baseline differences are within 1-2 points, it would be helpful to state whether the overall averages are computed across the same set of datasets for all systems.","section":"Section 5.3.1"},{"comment":"The claim that internalizing class definitions via 'import' improves efficiency is plausible, but no prompt-length or wall-clock efficiency measurement is reported in the experiments, even though Section 4.1.3 motivates this design by efficiency. A small table or figure quantifying prompt-length reduction would make the efficiency claim concrete.","section":"Section 4.1.3"}],"recommendation":"major_revision","confidential_remarks":"The IE and taxonomy expansion contributions are solid and could support a strong paper on universal information extraction with code-based ontologies. The deep-research report-generation claim, however, is the paper's stated headline and is currently supported only by an evaluation whose judge is partially inside the system being judged. If the authors can add an independent human or blinded LLM evaluation, release the dataset, and include ablations, the paper would be much stronger. If they cannot, they should either demote the report-generation experiment to a case study or remove the comparative claim against OpenAI Deep Research and Grok3. I also note that the paper's own Section 4.3 admits that training data for the reasoning phase is not yet generated, which further weakens the attribution of the report-generation results to the proposed model."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the KDR framework and KnowCoder-V2's unified code-generation design are a genuine step forward for knowledge-grounded deep research, and the IE/KBQA experiments are broad and mostly convincing. But the paper's headline claim—that KDR produces better reports than OpenAI Deep Research and Grok3—rests on an evaluation that is neither independent nor reproducible, and that claim should not be taken at face value.\n\nWhat's new: KnowCoder-V2 builds on KnowCoder-V1's code-based representation but adds an offline knowledge organization phase, import-based extraction (so class definitions are internalized in model parameters rather than repeated in prompts), and augmentation for robustness, events, and multilingual IE. The training recipe is sensible and the results on 26 IE benchmarks plus ontology expansion are consistently ahead of strong baselines. The KBQA result on WebQSP (84.7 Hits@1 without fine-tuning) is also solid. The paper is candid about its own limits: knowledge update is a simple merge heuristic, and the reasoning-phase training data is not yet released.\n\nThe soft spot is exactly where the stress-test points. The report-generation comparison (Table 6, Figure 1) uses a self-constructed, unpublished dataset, scored by DeepSeek-V3 and DeepSeek-R1, with no human evaluation or error analysis. Worse, Figure 3's caption says DeepSeek-R1 handles the parts of KDR not related to knowledge analysis. So one of the two judges is running part of the system being evaluated. That gives the scoring a real independence problem. The \"insight\" metric is especially suspect because KDR is explicitly designed to produce code-computation artifacts, and the judge may simply be rewarding that style. The 8.9 vs 7.7 average margin is therefore not independently established. It might be real, but this paper doesn't show it.\n\nOther issues are smaller: no released model or evaluation code, so the IE results cannot be reproduced yet; some tables have placeholder references; and the report evaluation lacks error bars entirely. None of these sink the IE contribution, which looks like solid systems engineering.\n\nBottom line: this deserves peer review, but a serious referee should insist on an independent evaluation of the report-generation claim—human judgments or a neutral judge on a public dataset—and on artifact release before the headline is accepted. I'd bring it to reading group as a case study in LLM-judge evaluation pitfalls, and I'd cite the IE and KBQA results if I were working on universal IE. Serious thinker: yes.","headline":"A credible IE/KBQA system whose headline deep-research report claim is undermined by a non-independent LLM judge.","tokens_in":25101,"tokens_out":2801,"would_cite":true,"duration_ms":29893,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Treating knowledge as Python code, a 14B model claims to beat larger deep-research systems on report quality.","keywords":["deep research","knowledge organization","knowledge computation","code generation","information extraction","ontology expansion","knowledge base question answering","report generation"],"falsifier":"Re-run the report-generation comparison on a pre-registered third-party dataset with expert human raters who check every cited number against its source; if the KnowCoder-V2 reports no longer beat the strongest commercial deep-research system on average quality or on insight, the paper's headline claim fails. A cheaper check is to verify whether the code-generated reports contain citation counts or bibliographic entries that do not exist in the source documents.","tokens_in":23975,"feed_emoji":"🧠","tokens_out":8156,"duration_ms":80340,"temperature":0.7,"pith_summary":"This paper argues that today's deep research systems fall short when the task requires insight, because they treat knowledge as web pages and reason by writing text. The proposed fix is to make knowledge computable: define an ontology as Python classes, convert source documents into Python objects during an offline organization phase, and then let the model generate and run analysis code over those objects while writing the report. KnowCoder-V2, a 14-billion-parameter code model fine-tuned for this, uses import statements to recall class schemas and writes code both for extraction and for computing results such as citation counts and publication distributions. On more than thirty datasets the model is competitive or better on taxonomy building, entity and relation extraction, event extraction, and knowledge-base question answering; in a report-generation test on a self-constructed dataset about scientist achievements, the full framework scores 8.9 average versus 7.7 for the strongest commercial deep-research baseline, with the biggest gap on the 'insight' axis.","feed_headline":"A 14B model beats bigger engines with code-generated knowledge","feed_subtitle":"Structuring knowledge as Python objects lifts report insight from 4-5 to 8.7 on the paper's test.","key_machinery":"The load-bearing mechanism is unified code generation built on a code-as-knowledge representation: ontology concepts are Python classes and knowledge instances are Python objects. Extraction becomes an import clause plus object instantiation; update is merging objects with the same name; reasoning is generating analysis code that runs over a retrieved search_results list. The framework surrounds this with a cycle of ontology search, code generation, instance query, and code execution, plus an evaluation agent that sends failed or unsatisfactory executions back through the loop. Ontology alignment uses the same code representation: candidate relations are retrieved by embedding concept definitions and then confirmed by an LLM reading the class code.","core_discovery":"KnowCoder-V2's central claim is that deep research systems gain a new capability when the same code-generation mechanism is used for both organizing knowledge and computing over it. The ontology is written as Python classes, and extracted facts become Python objects; extraction is object instantiation, knowledge update is code merge, and analysis is executable code run against a list called search_results. Because the class definitions live in the model's parameters, the model can recall a schema with a short import clause instead of receiving all definitions in the prompt, which removes the prompt-length bottleneck of earlier code-based extraction. The paper reports that this design outperforms or matches strong baselines on taxonomy expansion, English and Chinese information extraction, robustness to perturbations, and WebQSP question answering, and that the KDR framework's reports are rated above commercial and open deep-research baselines on all five dimensions, with the largest advantage in insight (8.7 versus 4.1–5.3).","pith_inferences":["The 'insight' advantage probably comes from verifiable computation: citation counts, distributions, and charts can be checked, whereas free-text summaries cannot, so code execution may reduce fabricated numbers in generated reports.","The two-phase design suggests a natural test for other domains: pick any field with a stable ontology, build the offline knowledge base once, and compare report insight against retrieval-only systems.","If import-based schema recall is the active mechanism, improvements in code-pretrained base models should transfer directly to better knowledge organization with no new architectural change; scaling the base model would test this.","The human-in-the-loop ontology construction and the name-merge update rule are the least automated parts; replacing them with learned update and alignment could determine how far the approach scales beyond curated topics."],"forward_implications":["Deep research no longer needs to treat each task as a fresh online search; shared domain knowledge can be organized once and reused across tasks.","Large-scale schemas stop being a prompt-length problem, because a model that has learned class definitions can activate them with an import statement.","Analyses that require counting, aggregating, comparing across timestamps, or charting become executable computations rather than prose guesswork.","A 14-billion-parameter fine-tuned model can match or exceed much larger commercial deep-research systems on report completeness, thoroughness, factuality, coherence, and insight, if the reported evaluations hold."],"supporting_citations":[{"why":"Supplies the premise that Python classes and objects can represent ontology concepts and instance knowledge, which KnowCoder-V2 extends.","marker":"Li et al. (2024)"},{"why":"Supplies the two-stage ontology-alignment recipe of embedding retrieval followed by LLM refinement, used in ontology expansion.","marker":"Zeng et al. (2024)"},{"why":"Supplies the robustness augmentation algorithms and the RUIE benchmark used to train and evaluate robustness.","marker":"Zhu et al. (2025)"},{"why":"Supplies the massive-type event extraction dataset and partitioning strategy used to strengthen event extraction.","marker":"Liu et al. (2025)"},{"why":"Supplies the cross-lingual parallel data pipeline used to build the multilingual information-extraction training corpus.","marker":"Zuo et al. (2024)"},{"why":"Supplies the base code model that KnowCoder-V2 is fine-tuned from.","marker":"Hui et al. (2024)"},{"why":"Supplies the reasoning model used to run non-knowledge parts of the KDR framework and as one of the two judges scoring reports.","marker":"Guo et al. (2025)"},{"why":"Supplies a deep-research baseline and the LLM-scoring evaluation setup for report generation.","marker":"Li et al. (2025b)"},{"why":"Supplies the knowledge-graph question-answering setup with ground-truth topic entities and subgraph extraction that KnowCoder-V2 follows.","marker":"Sun et al. (2023)"}],"fun_headline_variants":["Code as knowledge: 14B model clicks insight score to 8.7","Same code that extracts facts also runs the analysis","Deep research gets a knowledge boost: code-generated objects","14B KnowCoder-V2 turns classes into insights, beats bigger models"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The report-generation result assumes that scores given by two large-language-model judges, one of which powers parts of the same system, on a self-built and unpublished dataset about scientists are a valid measure of report quality and insight.","fun_headline_variants_meta":{"raw":{"variants":["Code as knowledge: 14B model clicks insight score to 8.7","Same code that extracts facts also runs the analysis","Deep research gets a knowledge boost: code-generated objects","14B KnowCoder-V2 turns classes into insights, beats bigger models"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000708,"raw_usage":{"total_tokens":3232,"prompt_tokens":1028,"completion_tokens":2204,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":644,"completion_tokens_details":{"reasoning_tokens":2131}},"tokens_in":644,"tokens_out":2204,"duration_ms":16424,"temperature":1.0,"reasoning_tokens":2131,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:47:23.464154+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the report-generation comparison on a pre-registered third-party dataset with expert human raters who check every cited number against its source; if the KnowCoder-V2 reports no longer beat the strongest commercial deep-research system on average quality or on insight, the paper's headline claim fails. A cheaper check is to verify whether the code-generated reports contain citation counts or bibliographic entries that do not exist in the source documents.","supporting_citations":[],"review_version":1}