{"id":"d6f9e82b-c655-448e-a6bb-4eaa3c517a00","arxiv_id":"2506.15241","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"The authors construct a character-relationship dataset from the First Four Histories and show that a fine-tuned Qwen model plus GraphRAG improves relation extraction and question-answering for classical Chinese text.","lead":"This paper builds a GraphRAG pipeline for extracting character relationships from four classical Chinese histories, and tests it on relation extraction and question answering. The authors report that a domain-adapted 14B model reaches F1=0.68 for relation extraction and that adding a knowledge graph raises a general model's F1 from 0.08 to 0.19 on the C-CLUE benchmark.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline C-CLUE claim conflates GraphRAG's overall QA pipeline with the relation-extraction task used to compute F1, so the 0.08→0.19 result is not a clean relation-extraction gain.","rationale":"The reader's weakest assumption focuses on data availability, split definitions, and label noise. Those are real concerns but they are generic to empirical NLP work. I agree with the conditional verdict, but I identify a more specific, task-level threat: the headline C-CLUE number is computed on an end-to-end QA pipeline whose output format, scoring procedure, and retrieval setup are never specified, so it is unclear whether the 0.08→0.19 delta measures relation extraction, QA answer generation, or the effect of giving the model access to gold-bearing graph triplets. This is consistent with the manuscript's own text: Section 4.1 describes the generator as producing natural-language text, Section 4.2 calls the comparison a QA system evaluation, and Section 2.5.1 defines F1 for relation triplets. The mismatch is visible internally and does not depend on external assumptions about data release. The paper does contain genuine contributions—a newly constructed historical ontology, a Chinese historical text corpus, a scoring-reward filtering pipeline, and a demonstration of domain-adapted models—and the non-headline claims (Tables 4 through 6 internal comparisons) are more stable, but the headline GraphRAG claim needs a sharper evaluation protocol. A concrete, low-cost check (an empty-graph control plus a strict-triplet-output re-scoring) can settle the concern, so CONDITIONAL is the appropriate verdict rather than REJECT or UNVERDICTED.","tokens_in":12926,"tokens_out":1843,"duration_ms":15991,"concrete_test":"Ask the authors to publish the C-CLUE evaluation script and the exact reference/answer format. Then re-run the DeepSeek+RAG system under an 'empty-graph' control (retrieval disabled, only the prompt template retained) and under a 'retrieved-irrelevant-triplets' control (retrieved triplets sampled from entities not present in the gold answer). If the F1 gap between DeepSeek baseline and DeepSeek+RAG persists in both controls, the gain is due to prompt engineering or generation style, not graph retrieval. If the gap vanishes, the reported 0.08→0.19 improvement is attributable to answer leakage through retrieval. Also recompute F1 after requiring the model to output strict (head, relation, tail) triplets, matching Table 6's task, to confirm the 11-point claim is computed on the same task.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central empirical claim is that 'DeepSeek integrated with GraphRAG improves F1 by 11% (0.08 to 0.19) on the open-domain C-CLUE relation extraction dataset.' But Table 9 is labeled a performance evaluation of the GraphRAG system, and the pipeline in Section 4.1 is an end-to-end QA system: Cypher query generation, alias expansion, vector retrieval of triplets, and prompt-based generation with DeepSeek. The F1 metric defined in Section 2.5.1 is for relation extraction, yet the system output is natural-language answers, not typed relation triplets. The paper never states how generated answers were tokenized, parsed, or aligned to reference triplets for F1 computation, nor whether the 500-sample evaluation set or the C-CLUE gold annotations were used as the reference. More importantly, the comparison 'DeepSeek 0.08' vs 'DeepSeek+RAG 0.19' is not an ablation of RAG: the baseline DeepSeek answers from parametric memory only, while the GraphRAG variant is allowed to retrieve the gold-supporting triplets from the knowledge graph constructed from the same corpus, which can artificially inflate the gain if the retrieval oracle leaks answer content. The claimed 11-point improvement therefore conflates retrieval access to the answer with generation quality, and the authors' own Section 4.2 describes the result as confirming 'knowledge alignment and hallucination suppression' without isolating the retrieval contribution. Since the dataset is not released and the evaluation protocol is unspecified, the headline number is not independently interpretable, let alone verifiable.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a GraphRAG framework for historical Chinese texts, combining chain-of-thought prompting, self-instruction, and process supervision to build a character-relationship knowledge graph from the \"First Four Histories\" corpus with minimal manual annotation. The authors then use this knowledge graph to augment a general-purpose LLM (DeepSeek) in a retrieval-augmented generation system. They report that a domain-specific model, Xunzi-Qwen1.5-14B, with Simplified Chinese input and chain-of-thought prompting, achieves the best relation extraction F1 of 0.68 on their annotated corpus, and that DeepSeek integrated with GraphRAG improves F1 from 0.08 to 0.19 on the open-domain C-CLUE relation extraction dataset, surpassing Xunzi-Qwen1.5-14B (F1=0.12). The paper also claims the framework reduces hallucinations and improves interpretability.","tokens_in":13250,"tokens_out":5519,"duration_ms":47652,"significance":"If the empirical claims hold, the work offers a low-resource pipeline for constructing domain-specific knowledge graphs from classical Chinese texts and demonstrates a concrete way to enhance general LLMs with graph-structured retrieval. The use of process supervision and a reward model for automated triplet filtering is an interesting design choice. The paper includes an external benchmark (C-CLUE) and makes falsifiable quantitative claims, which is a strength. However, the significance is currently limited by the lack of a released dataset and the under-specified evaluation protocol, which prevent external verification of the central results.","major_comments":[{"comment":"The F1 metric is defined in Section 2.5.1 for relation extraction, but the GraphRAG system described in Section 4.1 is an end-to-end QA pipeline that outputs natural-language answers, not typed relation triplets. The paper never states how the generated answers were converted to triples, tokenized, or aligned to reference triplets for F1 and ROUGE computation, nor what reference is used for the C-CLUE dataset. Without this mapping, the headline improvement (0.08 to 0.19) is not interpretable as a relation-extraction gain.","section":"§4.2 / Table 9"},{"comment":"The comparison between DeepSeek (0.08) and DeepSeek+RAG (0.19) is not a clean ablation of RAG: the baseline has no retrieval, while the RAG variant retrieves triplets from a knowledge graph constructed from the same corpus used for evaluation. If C-CLUE test queries overlap with the corpus or the knowledge graph contains answer-supporting triplets, the measured gain may reflect oracle-like access to the answer rather than an improvement in generation quality. The authors should either control for retrieval access (e.g., retrieve unrelated triplets) or report the retrieval contribution separately.","section":"§4.2 / Table 9"},{"comment":"The train/test splits for the fine-tuning and evaluation reported in Tables 4, 5, and 6 are not stated. Section 2.3.2 mentions a 9:1 split for the 5,000-sample instruction dataset, but it is unclear whether the SFT models in Table 6 were evaluated on held-out data or on the same data used for fine-tuning. The 500-sample evaluation dataset is mentioned in Section 3.2 but not linked to the 5,000-passage corpus or the 4,000-sample fine-tuning set. Without explicit split descriptions, the F1=0.68 result cannot be trusted as a generalization measure.","section":"§3, Tables 4–6"},{"comment":"The scoring reward model used to filter extracted triplets is Xunzi-Qwen1.5-14B, the same model family that generates the initial triplets in the automated pipeline. This creates a potential circularity: the filter may systematically favor the generator's characteristic errors rather than genuine extraction quality. The validation of the reward model relies on a single illustrative example (Table 8) and does not report agreement with human annotations or the correlation between reward scores and ground-truth quality. This is a load-bearing issue for the claimed reliability of the automated extraction process.","section":"§3.2 / §3.3"},{"comment":"There are internal numerical inconsistencies. In Section 4.2 the text says Xunzi-Qwen1.5-14B achieves F1=0.15 on C-CLUE, but Table 9 reports 0.12. In Section 3.1.3 the text says Xunzi-Qwen2-7B achieves F1=0.64, but Table 6 reports 0.55. These discrepancies in the central reported numbers undermine confidence in the reliability of the rest of the tables.","section":"§4.2 vs Text; §3.1.3 vs Table 6"}],"minor_comments":[{"comment":"The abstract says the DeepSeek+GraphRAG improvement is '11%' (0.08 to 0.19); this is an 11-percentage-point increase, not an 11% relative increase (which would be 137.5%). Please clarify the wording throughout.","section":"Abstract"},{"comment":"The text describes '15% increase' and '25% increase' in F1 scores for Simplified Chinese input; these are percentage-point differences (0.40 to 0.55 and 0.29 to 0.54), so they should be reported as percentage-point changes.","section":"§3.1.2"},{"comment":"The SikuBERT and BERT rows report F1 values but leave ROUGE-1, ROUGE-2, ROUGE-L, and SCORE blank. Either provide these values or explain why they are omitted.","section":"Table 6"},{"comment":"Equation (5) for ROUGE-2 says 'Number of overlapping unigrams' but should be 'overlapping bigrams'. Also, the term '0-COT' used in Table 4 is ambiguous; clarify whether it means zero-shot chain-of-thought.","section":"§2.5.2"},{"comment":"The C-CLUE dataset is not described or cited anywhere in the paper; please provide a reference and specify the task, the evaluation set size, and how the historical knowledge graph is relevant to this open-domain dataset.","section":"§4.2 and Data Availability"},{"comment":"The GraphRAG implementation details are incomplete: the top-K value for vector similarity retrieval, the prompt template for the generator, and the threshold for the 'unable to answer' fallback are not specified, which hinders reproducibility.","section":"§4.1"},{"comment":"The Data Availability statement says 'Not applicable,' yet the paper introduces a new manually annotated dataset of 5,000 passages and 5,000 reasoning chains. Since the evaluation depends on this unpublished corpus, the statement is contradictory and should be replaced with a data release plan or repository link.","section":"Data Availability"}],"recommendation":"major_revision","confidential_remarks":"The manuscript's central idea is relevant and interesting for the digital humanities and NLP community, but the evaluation protocol has serious gaps that must be addressed before publication. The 'Data Availability: Not applicable' statement is problematic given that a new dataset is a central contribution; the journal may want to enforce a data/code availability policy. Also note the numerical inconsistencies between the text and tables, which need careful proofreading."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know: this is a domain-application paper, not a methods paper, and its central quantitative claim—DeepSeek+RAG jumping from 0.08 to 0.19 F1 on C-CLUE—is not as clean as the abstract suggests. The system being evaluated is an end-to-end QA pipeline, while the F1 metric is defined for relation triplets; the paper never explains how natural-language answers were converted to triplets for scoring. That alone makes the headline number hard to interpret.\n\nWhat is genuinely new: a 5,000-passage annotated corpus of character relationships from the First Four Histories (not released, unfortunately), and a systematic comparison of prompt strategies, glyph inputs, and fine-tuned domain models (Xunzi series) for relation extraction. The process-supervision scoring rules in Table 7 and the automated extraction-scoring dual-model pipeline are a sensible recipe for low-resource historical knowledge graph construction. The case study in Figure 10 is illustrative rather than proof, but it does show what the system can do.\n\nThe good parts: the paper is honest about limitations, the ontology design is grounded in prior work, and using C-CLUE as an external benchmark for one comparison is the right instinct. The best result, Xunzi-Qwen1.5-14B at F1=0.68 on their own corpus, is plausible given the domain-specific fine-tuning, though without data release or split descriptions it's unverifiable.\n\nWhere it's soft: the evaluation protocol is underspecified. No train/test splits are stated for Tables 4-6; no error bars or significance tests anywhere; the reward model used for filtering is fine-tuned from the same base model family as the extraction model, creating a potential circularity; and the dataset's Data Availability says 'Not applicable,' which is unforgivable for a paper whose main contribution is a new annotated corpus. The stress-test concern about retrieval leakage is real: if the knowledge graph contains the same passages used to construct C-CLUE's gold answers, then the RAG variant may simply retrieve the answer rather than reason. The paper needs an ablation that separates retrieval from generation, and a clear description of how F1 was computed on QA outputs.\n\nWho this is for: researchers in digital humanities and Chinese NLP who want a template for building historical knowledge graphs with LLMs. They'll find the pipeline useful but should not rely on the numbers as published.\n\nRecommendation: send it to peer review, but only with the expectation of heavy revision. The authors should release the corpus, specify the evaluation protocol, add error bars, and re-analyze the C-CLUE comparison with a clean ablation. The core idea is worth engaging with; the evidence isn't.","headline":"A solid domain-application paper for classical-Chinese knowledge graph construction, but the headline GraphRAG gain on C-CLUE is not a clean relation-extraction result and the evaluation protocol needs substantial work.","tokens_in":13753,"tokens_out":2421,"would_cite":false,"duration_ms":23234,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A knowledge graph built from four classical Chinese histories improves a general LLM's relation extraction on an open benchmark, with F1 rising from 0.08 to 0.19 when GraphRAG is added.","keywords":["GraphRAG","knowledge graph","relation extraction","historical text","chain-of-thought prompting","process supervision","self-instruction","large language model"],"falsifier":"Run the relation-extraction and GraphRAG evaluations on a 500-sample test set that is explicitly disjoint from the 4,000 instruction-tuning samples, with the same scoring rubric; if Xunzi-Qwen1.5-14B no longer scores F1 ≈ 0.68 and DeepSeek+GraphRAG no longer scores F1 ≈ 0.19 on C-CLUE, the claimed advantage is an artifact of evaluation leakage or label noise.","tokens_in":12682,"feed_emoji":"📜","tokens_out":11234,"duration_ms":100512,"temperature":0.7,"pith_summary":"This paper claims that a mostly automated pipeline can turn classical Chinese historical texts into a character-relationship knowledge graph and that retrieving from that graph makes a general-purpose large language model measurably better at historical question answering. The anchor results are two F1 numbers: a domain-tuned model, Xunzi-Qwen1.5-14B, reaches F1 = 0.68 for relation extraction when prompted with chain-of-thought reasoning and simplified Chinese input, and adding graph-retrieval augmentation to DeepSeek raises its F1 from 0.08 to 0.19 on the open-domain C-CLUE benchmark, above Xunzi's 0.12. If these measurements are right, the framework offers a low-resource route to structured historical knowledge services and a partial check on hallucination in classical-text applications.","feed_headline":"History knowledge graph lifts relation F1 from 0.08 to 0.19","feed_subtitle":"Graph-retrieved triplets feed DeepSeek, raising classical-Chinese relation extraction by 11 F1 points on C-CLUE.","key_machinery":"The mechanism that carries the argument is a five-step chain-of-thought extraction scaffold paired with a scoring filter and a graph-augmented generator. The extraction scaffold walks a model through translation into modern Chinese, named entity recognition, semantic relation analysis, relationship-schema construction, and triplet extraction; using this scaffold, a self-instruction-generated training set, and simplified-Chinese input, Xunzi-Qwen1.5-14B produces candidate triplets. A second score-and-reward model applies a ten-point penalty rubric to each reasoning step and keeps only high-scoring triplets, which are loaded into a graph database as the historical character-relationship knowledge graph. At query time, DeepSeek extracts entities from the question, expands them through an alias dictionary, retrieves candidate triplets with structured graph queries plus vector similarity, and passes readable triplets into the prompt as prior knowledge, refusing to answer when the graph has no evidence.","core_discovery":"On the paper's own terms, the central discovery is that graph-structured memory can substitute for expensive domain retraining. The authors report that chain-of-thought prompting and simplified-Chinese input push the domain-adapted Xunzi-Qwen1.5-14B to F1 = 0.68 on their First Four Histories relation-extraction task, far above GPT-4o's 0.28 and DeepSeek's 0.23. When DeepSeek is paired with the constructed knowledge graph through graph retrieval and alias-expanded entity linking, its C-CLUE F1 rises from 0.08 to 0.19, passing Xunzi-Qwen1.5-14B's 0.12 and giving the graph-augmented general model the best open-domain score in the paper. The authors take this as evidence that the knowledge graph supplies explicit multi-hop relational paths, reducing hallucinations and making answers more interpretable.","pith_inferences":["Because the paper does not state the train/test split for its evaluation tables, an independent rerun on a released 500-sample holdout is the direct test of whether the F1 gains are real or an artifact of leakage.","The explicit refusal rule ('unable to answer') probably accounts for part of the hallucination drop; separating abstention from correct answers would show how much of the gain is new knowledge rather than silence.","The paper's own limitations—coverage of only the Four Histories and possible annotation gaps—imply the framework's generality to other classical corpora is still open; transferring it would need fresh annotation and a new held-out benchmark."],"forward_implications":["General-purpose models can be made useful for classical Chinese knowledge tasks by attaching a domain knowledge graph, without retraining the underlying model.","The combination of self-instruction data and process-supervision scoring can produce usable relation-extraction training sets from a relatively small manually annotated corpus.","Adding graph retrieval and an explicit refusal rule gives users a way to trace an answer to supporting relational evidence, which bears directly on hallucination in historical question answering.","The same extraction-and-filter pipeline should be reusable for other classical corpora, though the paper's own experiments only cover the First Four Histories."],"supporting_citations":[{"why":"Supplies the retrieval-augmented generation paradigm that the GraphRAG system builds on.","marker":"Lewis et al. 2020"},{"why":"Grounds the chain-of-thought prompt strategy used for relation extraction.","marker":"Wei et al. 2023"},{"why":"Provides the Self-Instruct method used to generate the 5,000-chain CoT training set with minimal manual annotation.","marker":"Wang et al. 2023"},{"why":"Supplies the automated process-supervision idea behind the five-stage scoring reward model.","marker":"Luo et al. 2024"},{"why":"Defines the broader GraphRAG framework that the paper adapts to historical texts.","marker":"Peng et al. 2024"},{"why":"Provides the multi-dimensional historical-text ontology model reused for character relationships.","marker":"Zhang et al. 2022"}],"fun_headline_variants":["Graph RAG boosts DeepSeek F1 from 0.08 to 0.19","Graph retrieval beats fine-tuning for classical Chinese relations","Graph-augmented DeepSeek outdoes domain-tuned Xunzi on history","Knowledge graph turns general LLM into history expert","DeepSeek plus graph memory beats training on historical text"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The F1 comparisons assume the 5,000-passage corpus and the 500-sample evaluation set are accurate ground truth and that the models were tested on data they were not fine-tuned on, but the paper does not state the split or release the data.","fun_headline_variants_meta":{"raw":{"variants":["Graph RAG boosts DeepSeek F1 from 0.08 to 0.19","Graph retrieval beats fine-tuning for classical Chinese relations","Graph-augmented DeepSeek outdoes domain-tuned Xunzi on history","Knowledge graph turns general LLM into history expert","DeepSeek plus graph memory beats training on historical text"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00067,"raw_usage":{"total_tokens":3065,"prompt_tokens":969,"completion_tokens":2096,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":585,"completion_tokens_details":{"reasoning_tokens":2005}},"tokens_in":585,"tokens_out":2096,"duration_ms":15019,"temperature":1.0,"reasoning_tokens":2005,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:39:18.808266+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the relation-extraction and GraphRAG evaluations on a 500-sample test set that is explicitly disjoint from the 4,000 instruction-tuning samples, with the same scoring rubric; if Xunzi-Qwen1.5-14B no longer scores F1 ≈ 0.68 and DeepSeek+GraphRAG no longer scores F1 ≈ 0.19 on C-CLUE, the claimed advantage is an artifact of evaluation leakage or label noise.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the retrieval-augmented generation paradigm that the GraphRAG system builds on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the broader GraphRAG framework that the paper adapts to historical texts."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the multi-dimensional historical-text ontology model reused for character relationships."}],"review_version":2}