{"id":"3924c0eb-7e85-485a-b005-9604c7dad2d3","arxiv_id":"2411.11090","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"ForPKG builds a forestry policy knowledge graph with a fine-grained ontology and an LLM-based extraction pipeline, reporting 76.2% precision and 62.6% recall on 50 annotated documents.","lead":"ForPKG is a framework that reads Chinese forestry policy documents and turns them into a knowledge graph using a new 10-type, 15-relation ontology and a three-step large language model pipeline. The authors report much higher extraction accuracy than off-the-shelf tools and show the graph improves a retrieval-augmented question answerer, but the evaluation lacks key validation details.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table III's headline numbers depend on an unstated train/evaluation split; if the 150 prompt-learning examples overlap the 50 evaluated documents, the 76.2%/62.6% result overstates general extraction performance.","rationale":"I read the paper as making a concrete empirical claim: the proposed three-step extraction pipeline outperforms off-the-shelf and direct-LLM baselines on forestry policy triples. The ontology and framework contributions are real and the pipeline description is reasonably specific, but the evidence for the headline number is the load-bearing part. The reader's weakest assumption and my own analysis converge on the same point: the relation classifier in Section IV-C2 is trained on 150 human-annotated examples, and Section V-B1 evaluates on 50 human-annotated documents, but no train/test separation is documented. The manuscript text does not rule out leakage, and without a clear statement or released data split, the 76.2%/62.6% figures cannot be interpreted as evidence of generalization. I agree with the reader's verdict and see no need to change it; the concern is concrete and checkable, and it would be settled by inspecting the repository. I would flag the secondary issues (the 'unsupervised' label, one-round gold annotation, no IAA) as revision requirements, but the single decisive check is whether the 150 training samples and the 50 test documents overlap.","tokens_in":10547,"tokens_out":3677,"duration_ms":105976,"concrete_test":"Download the released GitHub repository and inspect the data files associated with Algorithm 1 (the 150 training samples) and Section V-B1 (the 50 annotated documents and 1,126 gold triples). Compute the overlap at the sentence level: normalize Chinese text and check whether any training sample sentence is contained in or near-duplicate of a sentence in the 50 evaluation documents, and whether any gold triple's head or tail appears verbatim in the training data. Report the overlap count and file provenance. If the overlap is nonzero, Table III cannot support the abstract's claim and the evaluation must be redone with a disjoint train/test split; if the overlap is zero, this specific concern is resolved and the performance numbers are much more credible.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim is the Table III comparison, where the proposed three-step pipeline reaches 76.2% precision and 62.6% recall on 1,126 manually annotated triples from 50 forestry policy documents, versus at most 50.7%/23.3% for direct LLM baselines. Step 2 of the pipeline (Section IV-C2) fine-tunes a DeBERTa relation classifier using 150 training samples, 'carefully constructed' at 10 examples per relationship type. Section V-B1 then evaluates on 50 documents that were 'manually annotated' for triples. Nowhere does the paper state that the 150 training samples are disjoint from the 50 evaluation documents, nor that the training triples were not drawn from those same documents. If the training samples come from the same documents or share sentences with the gold triples, the relation classifier can memorize the identity of the head entity and text pattern, and the reported 76.2%/62.6% would not reflect generalization to unseen policy text. This is not a cosmetic issue: the paper's abstract claims superiority over other unsupervised methods, and that claim rests entirely on this unstated split. The related labeling issue, that the method is called 'unsupervised' despite using 150 labeled examples, is secondary; the first-order question is whether the labeled examples are independent of the test set.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ForPKG, a framework for constructing a forestry policy knowledge graph. It defines a 10-type/15-relation ontology, a three-step extraction pipeline (Kimi-based head-entity recognition, DeBERTa prompt-learning relation classification, Kimi-based tail-entity recognition), and reports precision/recall on triple extraction from 50 Chinese forestry policy documents. It also presents a RAG application evaluation scored by GPT-4 and announces that the knowledge graph will be released on GitHub. The paper's main empirical claim is that the proposed extraction method outperforms unsupervised off-the-shelf tools and direct LLM baselines, with Table III reporting 76.2% precision and 62.6% recall.","tokens_in":10723,"tokens_out":5602,"duration_ms":55575,"significance":"If the reported extraction numbers were valid, the paper would be a useful contribution to domain-specific knowledge graph construction: the ontology is grounded in deontic logic, the pipeline is concrete and reproducible in principle, and the planned public release of the knowledge graph is a genuine community resource. The comparison against several off-the-shelf extraction tools and direct LLM baselines is a reasonable starting point. However, the central empirical claim is not currently supported because the paper does not separate the supervision source from the evaluation set, mischaracterizes the method as unsupervised, and provides no statistical or agreement evidence. As it stands, the contribution is a framework proposal whose headline quantitative results require substantial additional validation.","major_comments":[{"comment":"The paper never states that the 150 prompt-learning training samples (10 per relationship type) used to fine-tune DeBERTa are disjoint from the 50 documents whose 1,126 triples are used to compute Table III. If the training examples and evaluation documents overlap, the relation classifier can memorize entity/pattern associations, and the 76.2%/62.6% headline figures would overstate generalization. The authors must state the split explicitly and, if necessary, re-run the evaluation on held-out documents.","section":"Section IV-C2; Section V-B1; Table III"},{"comment":"The method is called 'unsupervised' in the abstract and in Section IV, but Section IV-C2 fine-tunes DeBERTa with 150 manually labeled examples. This makes the comparison against DeepKE, LTP, Jiagu, and DeepDive, described as 'unsupervised,' not apples-to-apples. The method should be described as few-shot or lightweight-supervised, and the baselines should include supervised or few-shot alternatives; otherwise the reported margin could be attributed to the use of labels rather than to the proposed architecture.","section":"Abstract; Section IV-C2"},{"comment":"No variance is reported for any row of Table III, and no inter-annotator agreement is given for the manually annotated gold triples. LLM API outputs are stochastic, so point estimates without repeated runs or confidence intervals do not support the claim that the improvement over the strongest direct-LLM baseline (50.7%/23.3%) is significant. The authors should report means and standard deviations over multiple runs, and at least one agreement metric for the gold annotation.","section":"Section V-B1; Table III"},{"comment":"The ontology extensibility experiment is not described in sufficient detail: the text does not state how many queries were used, what the 'response accuracy' measure is, or how the fusion with OwnThink, OpenKG, CN-Dbpedia, DBpedia, and OpenConcept was performed. Without this information, Figure 2 cannot be interpreted and the claim that the ontology has 'good expressiveness and extensibility' is not supported.","section":"Section V-A; Figure 2"},{"comment":"The RAG evaluation uses GPT-4 as the scorer but reports no number of test questions, no sample prompts, no validation of the GPT-4 scoring, and no statistical test or confidence intervals. The claim that 'the practical application value of the knowledge graph ... is confirmed' goes beyond what this evidence can establish. The authors should provide the test set, scoring template, agreement between GPT-4 and human raters, and significance testing.","section":"Section V-C; Figure 5"}],"minor_comments":[{"comment":"The formula uses a slash where a conditional-probability bar is intended; write log P([MASK] = phi(y_i) | T(s_i, h_i)).","section":"Algorithm 1, line 6"},{"comment":"Table II contains the typos 'Iinverse Relationship' and 'Contrain'; also, the 'relevant' relation is described as reflexive, but the construction via pairwise cosine similarity yields an undirected/symmetric relation, not a reflexive one.","section":"Table II; Section IV-B"},{"comment":"The Chinese checkpoint of BigBird is mentioned without a citation; provide the source or model identifier so the encoding step is reproducible.","section":"Section IV-B"},{"comment":"The 'accuracy rates' shown in the radar chart should be defined as precision, recall, or F1, and the number of instances per entity/relationship type should be reported, because the figure is hard to interpret without denominators.","section":"Section V-B2; Figure 3"},{"comment":"The GitHub repository is mentioned in the abstract, but the experimental section does not state which materials (documents, annotations, fine-tuned model, or evaluation scripts) are actually included; please clarify the release contents at the point where the experiments are described.","section":"Abstract; Section VII"}],"recommendation":"major_revision","confidential_remarks":"The central empirical claim rests on the missing train/test split and the 'unsupervised' mischaracterization. If the authors can provide a clean held-out evaluation and a more honest framing, the framework may be publishable as a domain-specific knowledge graph construction case study. I would also ask the editor to weigh whether the two 'first' novelty claims (first forestry policy knowledge graph and first fine-grained forestry policy ontology) are sufficiently scoped, since the related-work section is brief and does not fully survey existing policy knowledge graph efforts."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper's actual contribution is the fine-grained forestry policy ontology and the promise of a released KG, not the extraction numbers. The three-stage pipeline (Kimi head-entity recognition, DeBERTa prompt-learning relation classification, Kimi tail-entity recognition) is a reasonable configuration for long abstract entities, and the deontic relationship types (duty, prohibition, right) are a genuine domain modeling choice. That deserves credit.\n\nThe soft spot is load-bearing. Table III reports 76.2% precision and 62.6% recall against a manual gold standard, and the abstract claims superiority over other unsupervised methods. But the method is not unsupervised: it fine-tunes a DeBERTa classifier on 150 manually constructed examples. And the paper never states that those 150 training samples are disjoint from the 50 documents used for the Table III evaluation. If they overlap, the relation classifier can memorize head-entity patterns and the headline numbers overstate generalization. That is the first question a referee should ask, and the manuscript as written cannot answer it.\n\nSecondary issues reinforce the concern. Kimi is called open-source but it is a proprietary API. No inter-annotator agreement is reported for the gold triples, and no error bars appear anywhere. The RAG evaluation uses GPT-4 to score LLaMa-Chinese outputs on correctness, effectiveness, and fluency, with no human validation. The document-similarity threshold lambda is a free parameter with no sensitivity analysis. None of these are fatal by themselves; together they mean the empirical support is thinner than the abstract implies.\n\nThe ontology fusion experiment (Figure 2) is also vague: it reports improved response accuracy after fusing with five general KGs, but it does not describe the queries, the baseline, or how accuracy is measured.\n\nBottom line: this is a resource paper with a useful ontology and a plausible pipeline, but the central empirical claim is not supported as reported. It deserves a serious referee, not a desk reject, because the framework and resource are real and the authors can fix the issues with a clear train/test split, released artifacts, honest method labeling, and a human-validated RAG evaluation. I would not cite the numbers until that is done, but I would bring it to a reading group as an example of what resource-paper evaluation needs to get right.","headline":"The ontology and the promise of a released KG are the real contribution; the headline extraction numbers are plausible but unsupported because the paper never states that the 150 training samples and the 50 evaluation documents are disjoint.","tokens_in":11334,"tokens_out":2190,"would_cite":false,"duration_ms":21811,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims a three-step pipeline using an open-source large language model can extract forestry policy triples with 76.2% precision and 62.6% recall, beating unsupervised baselines.","keywords":["knowledge graph construction","policy knowledge graph","forestry policy","ontology design","information extraction","prompt learning","large language models","retrieval-augmented generation"],"falsifier":"Re-run the same three-step pipeline on 50 held-out forestry policy documents whose texts were never used to build the 150 relationship-classification examples, and compare precision and recall against Table III; if the numbers fall sharply, the published estimate depends on training/test overlap. A second independent annotator re-marking the same 50 documents would also test whether the 1,126 triples are stable enough to serve as a gold standard.","tokens_in":10237,"feed_emoji":"🌲","tokens_out":8954,"duration_ms":76372,"temperature":0.7,"pith_summary":"This paper argues that a usable forestry policy knowledge graph can be built without large labeled datasets by combining a fine-grained ontology with a decomposed open-source large-language-model pipeline. The ontology defines 10 entity types and 15 relationship types, including normative relations such as duty, prohibition, and right drawn from deontic logic. The extraction pipeline first uses an LLM to find head entities, then a prompt-learned text classifier to pick the relation, then the LLM to find the tail entity; on 1,126 manually annotated triples from 50 Chinese forestry policy documents it reports 76.2% precision and 62.6% recall, above direct LLM calls and off-the-shelf extraction frameworks. The authors also show that when the resulting graph is used as a retrieval source for a large language model, answer quality improves, which is the practical payoff they care about.","feed_headline":"LLM pipeline extracts forestry policy triples at 76.2% precision","feed_subtitle":"A graph built from 757 documents beats direct large-model and off-the-shelf baselines on 1,126 annotated triples.","key_machinery":"The load-bearing machinery is the three-step content-level extraction process: (1) Kimi, an open-source Chinese LLM, is prompted to recognize head entities from the full policy document; (2) a DeBERTa model fine-tuned with prompt learning, using relation names as the verbalizer, classifies which of the 15 ontology relations applies; (3) Kimi locates the relation word and extracts the tail entity. Step order matters because long tail entities tend to follow the relation word, so most of the difficult span-detection is deferred to a pattern-assisted step, and the abstract relations are converted into a bounded classification task. This pipeline is what carries the reported performance advantage and what makes the framework transferable to other policy domains.","core_discovery":"The central claim is that the hard part of policy knowledge extraction is not language understanding but division of labour: asking one model to produce complete triples in one pass fails, while splitting the job into head-entity recognition, relation classification, and tail-entity recognition succeeds on long abstract normative entities like '退耕还林还草补助资金兑付' and on relations such as obligation and prohibition. The paper further claims that relation classification is best treated as a few-shot text classification problem, with 150 hand-built examples and a prompt-learning-fine-tuned DeBERTa model, rather than as an open-ended LLM generation task. On its test corpus, this design gives 76.2% precision and 62.6% recall, and the paper argues this confirms the ontology is expressive and extensible enough to support both direct policy applications and retrieval-augmented generation.","pith_inferences":["The paper never explicitly separates the 150 prompt-learning training examples from the 50 evaluation documents, so the 76.2%/62.6% figures should be re-audited on fully held-out documents before the comparison is taken at face value.","Since tail-entity extraction relies on finding the relation word, sentences that express obligations or definitions without an explicit cue word are likely to be the pipeline's weak spot; a stress test with cue-poor sentences would reveal the true ceiling.","Calling the method unsupervised is softened by 150 hand-built examples; a zero-shot variant that omits the DeBERTa step would test which part of the pipeline is doing the work.","The application analysis depends on an LLM judge rather than human raters, so a small human evaluation would make the RAG improvement claim easier to rely on."],"forward_implications":["A policy knowledge graph that reaches inside the text, rather than just linking documents to metadata, can support compliance checking, policy analysis, and intelligent policy question answering.","The extraction recipe needs only about 150 labeled examples, so other low-resource policy fields can imitate it without building large annotated corpora.","Feeding the graph into a retrieval-augmented generation system lifts the generator's correctness, effectiveness, and fluency scores in the paper's evaluation.","The ontology's 10 entity types and 15 relationship types can be merged with general knowledge graphs, and the paper reports improved query response accuracy after fusion.","Because the ontology and pipeline are not forestry-specific, the same framework is intended to transfer to healthcare, insurance, and finance policy text."],"supporting_citations":[{"why":"Supplies the deontic-logic theory behind the three normative relationship types (duty, prohibition, right) that make the ontology policy-specific.","marker":"[24]"},{"why":"Provides the few-shot prompt-learning method used to train the relation classifier from 150 examples.","marker":"[28]"},{"why":"Supplies one of the off-the-shelf extraction framework baselines in the Table III comparison.","marker":"[29]"},{"why":"Supplies another Chinese-language processing baseline in the triple-extraction comparison.","marker":"[30]"},{"why":"Supplies a further knowledge-extraction system baseline in the triple-extraction comparison.","marker":"[31]"},{"why":"Supplies the open-source Chinese large language model used both as a direct-extraction baseline and as the generator in the RAG application test.","marker":"[26]"}],"fun_headline_variants":["ForPKG: split LLM tasks to extract forestry policy triples","Division of labor improves forestry policy knowledge graph construction","Unsupervised framework builds forestry policy graphs with better precision","Ontology and split extraction yield 76.2% precision on policy triples"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The 150 relationship-classification examples used to train the prompt-learning classifier are never explicitly separated from the 50 documents used to measure precision and recall, and the authors' manual annotation is the only gold standard; if those overlap or the annotation is unreliable, the reported numbers do not measure real extraction performance.","fun_headline_variants_meta":{"raw":{"variants":["ForPKG: split LLM tasks to extract forestry policy triples","Division of labor improves forestry policy knowledge graph construction","Unsupervised framework builds forestry policy graphs with better precision","Ontology and split extraction yield 76.2% precision on policy triples"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000223,"raw_usage":{"total_tokens":1473,"prompt_tokens":974,"completion_tokens":499,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":590,"completion_tokens_details":{"reasoning_tokens":426}},"tokens_in":590,"tokens_out":499,"duration_ms":5787,"temperature":1.0,"reasoning_tokens":426,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T18:55:05.359087+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the same three-step pipeline on 50 held-out forestry policy documents whose texts were never used to build the 150 relationship-classification examples, and compare precision and recall against Table III; if the numbers fall sharply, the published estimate depends on training/test overlap. A second independent annotator re-marking the same 50 documents would also test whether the 1,126 triples are stable enough to serve as a gold standard.","supporting_citations":[{"cited_title":"Deontic logic,","cited_arxiv_id":null,"evidence_quote":"Supplies the deontic-logic theory behind the three normative relationship types (duty, prohibition, right) that make the ontology policy-specific."},{"cited_title":"DeepKE: A Deep Learning Based Knowledge Extraction Toolkit for Knowledge Base Population","cited_arxiv_id":"2201.03335","evidence_quote":"Supplies one of the off-the-shelf extraction framework baselines in the Table III comparison."},{"cited_title":"N-LTP: An Open-source Neural Language Technology Platform for Chinese","cited_arxiv_id":"2009.11616","evidence_quote":"Supplies another Chinese-language processing baseline in the triple-extraction comparison."},{"cited_title":"Deepdive: Web-scale knowledge-base construction using statistical learning and inference","cited_arxiv_id":null,"evidence_quote":"Supplies a further knowledge-extraction system baseline in the triple-extraction comparison."}],"review_version":1}