Pith. sign in

REVIEW 5 major objections 7 minor 30 references

Research on Graph-Retrieval Augmented Generation Based on Historical Text Knowledge Graphs

T0 review · 5 major / 7 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A knowledge graph built from four classical Chinese histories improves a general LLM's relation extraction on an open benchmark, with F1 rising from 0.08 to 0.19 when GraphRAG is added.

desk verdict A solid domain-application paper for classical-Chinese knowledge graph construction, but the headline GraphRAG gain on C-CLUE is not a clean relation-extraction result and the evaluation protocol needs substantial work. read the letter →

arxiv 2506.15241 v1 pith:W6KKEHSG submitted 2025-06-18 cs.CL

Yang Fan , Zhang Qi , Xing Wenqian , Liu Chang , Liu Liu This is my paper · ORCID
classification cs.CL
keywords GraphRAGknowledgerelationextractionhistoricaltextchain-of-thoughtpromptingprocesssupervisionself-instructionlargelanguagemodel
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a mostly automated pipeline can turn classical Chinese historical texts into a character-relationship knowledge graph and that retrieving from that graph makes a general-purpose large language model measurably better at historical question answering. The anchor results are two F1 numbers: a domain-tuned model, Xunzi-Qwen1.5-14B, reaches F1 = 0.68 for relation extraction when prompted with chain-of-thought reasoning and simplified Chinese input, and adding graph-retrieval augmentation to DeepSeek raises its F1 from 0.08 to 0.19 on the open-domain C-CLUE benchmark, above Xunzi's 0.12. If these measurements are right, the framework offers a low-resource route to structured historical knowledge services and a partial check on hallucination in classical-text applications.

What carries the argument

The mechanism that carries the argument is a five-step chain-of-thought extraction scaffold paired with a scoring filter and a graph-augmented generator. The extraction scaffold walks a model through translation into modern Chinese, named entity recognition, semantic relation analysis, relationship-schema construction, and triplet extraction; using this scaffold, a self-instruction-generated training set, and simplified-Chinese input, Xunzi-Qwen1.5-14B produces candidate triplets. A second score-and-reward model applies a ten-point penalty rubric to each reasoning step and keeps only high-scoring triplets, which are loaded into a graph database as the historical character-relationship knowledge graph. At query time, DeepSeek extracts entities from the question, expands them through an alias dictionary, retrieves candidate triplets with structured graph queries plus vector similarity, and passes readable triplets into the prompt as prior knowledge, refusing to answer when the graph has no evidence.

What would settle it

Run the relation-extraction and GraphRAG evaluations on a 500-sample test set that is explicitly disjoint from the 4,000 instruction-tuning samples, with the same scoring rubric; if Xunzi-Qwen1.5-14B no longer scores F1 ≈ 0.68 and DeepSeek+GraphRAG no longer scores F1 ≈ 0.19 on C-CLUE, the claimed advantage is an artifact of evaluation leakage or label noise.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that graph-structured memory can substitute for expensive domain retraining. The authors report that chain-of-thought prompting and simplified-Chinese input push the domain-adapted Xunzi-Qwen1.5-14B to F1 = 0.68 on their First Four Histories relation-extraction task, far above GPT-4o's 0.28 and DeepSeek's 0.23. When DeepSeek is paired with the constructed knowledge graph through graph retrieval and alias-expanded entity linking, its C-CLUE F1 rises from 0.08 to 0.19, passing Xunzi-Qwen1.5-14B's 0.12 and giving the graph-augmented general model the best open-domain score in the paper. The authors take this as evidence that the knowledge graph supplies explicit multi-hop relational paths, reducing hallucinations and making answers more interpretable.

Load-bearing premise

The F1 comparisons assume the 5,000-passage corpus and the 500-sample evaluation set are accurate ground truth and that the models were tested on data they were not fine-tuned on, but the paper does not state the split or release the data.

Editorial extensions

If this is right

  • General-purpose models can be made useful for classical Chinese knowledge tasks by attaching a domain knowledge graph, without retraining the underlying model.
  • The combination of self-instruction data and process-supervision scoring can produce usable relation-extraction training sets from a relatively small manually annotated corpus.
  • Adding graph retrieval and an explicit refusal rule gives users a way to trace an answer to supporting relational evidence, which bears directly on hallucination in historical question answering.
  • The same extraction-and-filter pipeline should be reusable for other classical corpora, though the paper's own experiments only cover the First Four Histories.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the paper does not state the train/test split for its evaluation tables, an independent rerun on a released 500-sample holdout is the direct test of whether the F1 gains are real or an artifact of leakage.
  • The explicit refusal rule ('unable to answer') probably accounts for part of the hallucination drop; separating abstention from correct answers would show how much of the gain is new knowledge rather than silence.
  • The paper's own limitations—coverage of only the Four Histories and possible annotation gaps—imply the framework's generality to other classical corpora is still open; transferring it would need fresh annotation and a new held-out benchmark.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 7 minor

Summary. The paper proposes a GraphRAG framework for historical Chinese texts, combining chain-of-thought prompting, self-instruction, and process supervision to build a character-relationship knowledge graph from the "First Four Histories" corpus with minimal manual annotation. The authors then use this knowledge graph to augment a general-purpose LLM (DeepSeek) in a retrieval-augmented generation system. They report that a domain-specific model, Xunzi-Qwen1.5-14B, with Simplified Chinese input and chain-of-thought prompting, achieves the best relation extraction F1 of 0.68 on their annotated corpus, and that DeepSeek integrated with GraphRAG improves F1 from 0.08 to 0.19 on the open-domain C-CLUE relation extraction dataset, surpassing Xunzi-Qwen1.5-14B (F1=0.12). The paper also claims the framework reduces hallucinations and improves interpretability.

Significance. If the empirical claims hold, the work offers a low-resource pipeline for constructing domain-specific knowledge graphs from classical Chinese texts and demonstrates a concrete way to enhance general LLMs with graph-structured retrieval. The use of process supervision and a reward model for automated triplet filtering is an interesting design choice. The paper includes an external benchmark (C-CLUE) and makes falsifiable quantitative claims, which is a strength. However, the significance is currently limited by the lack of a released dataset and the under-specified evaluation protocol, which prevent external verification of the central results.

major comments (5)
  1. [§4.2 / Table 9] The F1 metric is defined in Section 2.5.1 for relation extraction, but the GraphRAG system described in Section 4.1 is an end-to-end QA pipeline that outputs natural-language answers, not typed relation triplets. The paper never states how the generated answers were converted to triples, tokenized, or aligned to reference triplets for F1 and ROUGE computation, nor what reference is used for the C-CLUE dataset. Without this mapping, the headline improvement (0.08 to 0.19) is not interpretable as a relation-extraction gain.
  2. [§4.2 / Table 9] The comparison between DeepSeek (0.08) and DeepSeek+RAG (0.19) is not a clean ablation of RAG: the baseline has no retrieval, while the RAG variant retrieves triplets from a knowledge graph constructed from the same corpus used for evaluation. If C-CLUE test queries overlap with the corpus or the knowledge graph contains answer-supporting triplets, the measured gain may reflect oracle-like access to the answer rather than an improvement in generation quality. The authors should either control for retrieval access (e.g., retrieve unrelated triplets) or report the retrieval contribution separately.
  3. [§3, Tables 4–6] The train/test splits for the fine-tuning and evaluation reported in Tables 4, 5, and 6 are not stated. Section 2.3.2 mentions a 9:1 split for the 5,000-sample instruction dataset, but it is unclear whether the SFT models in Table 6 were evaluated on held-out data or on the same data used for fine-tuning. The 500-sample evaluation dataset is mentioned in Section 3.2 but not linked to the 5,000-passage corpus or the 4,000-sample fine-tuning set. Without explicit split descriptions, the F1=0.68 result cannot be trusted as a generalization measure.
  4. [§3.2 / §3.3] The scoring reward model used to filter extracted triplets is Xunzi-Qwen1.5-14B, the same model family that generates the initial triplets in the automated pipeline. This creates a potential circularity: the filter may systematically favor the generator's characteristic errors rather than genuine extraction quality. The validation of the reward model relies on a single illustrative example (Table 8) and does not report agreement with human annotations or the correlation between reward scores and ground-truth quality. This is a load-bearing issue for the claimed reliability of the automated extraction process.
  5. [§4.2 vs Text; §3.1.3 vs Table 6] There are internal numerical inconsistencies. In Section 4.2 the text says Xunzi-Qwen1.5-14B achieves F1=0.15 on C-CLUE, but Table 9 reports 0.12. In Section 3.1.3 the text says Xunzi-Qwen2-7B achieves F1=0.64, but Table 6 reports 0.55. These discrepancies in the central reported numbers undermine confidence in the reliability of the rest of the tables.
minor comments (7)
  1. [Abstract] The abstract says the DeepSeek+GraphRAG improvement is '11%' (0.08 to 0.19); this is an 11-percentage-point increase, not an 11% relative increase (which would be 137.5%). Please clarify the wording throughout.
  2. [§3.1.2] The text describes '15% increase' and '25% increase' in F1 scores for Simplified Chinese input; these are percentage-point differences (0.40 to 0.55 and 0.29 to 0.54), so they should be reported as percentage-point changes.
  3. [Table 6] The SikuBERT and BERT rows report F1 values but leave ROUGE-1, ROUGE-2, ROUGE-L, and SCORE blank. Either provide these values or explain why they are omitted.
  4. [§2.5.2] Equation (5) for ROUGE-2 says 'Number of overlapping unigrams' but should be 'overlapping bigrams'. Also, the term '0-COT' used in Table 4 is ambiguous; clarify whether it means zero-shot chain-of-thought.
  5. [§4.2 and Data Availability] The C-CLUE dataset is not described or cited anywhere in the paper; please provide a reference and specify the task, the evaluation set size, and how the historical knowledge graph is relevant to this open-domain dataset.
  6. [§4.1] The GraphRAG implementation details are incomplete: the top-K value for vector similarity retrieval, the prompt template for the generator, and the threshold for the 'unable to answer' fallback are not specified, which hinders reproducibility.
  7. [Data Availability] The Data Availability statement says 'Not applicable,' yet the paper introduces a new manually annotated dataset of 5,000 passages and 5,000 reasoning chains. Since the evaluation depends on this unpublished corpus, the statement is contradictory and should be replaced with a data release plan or repository link.

Circularity Check

1 steps flagged · score 4.0 of 10

Filter-reliability claim is self-validated by the reward model's own scores; main external claims remain independent.

  1. self definitional [Section 3.3, Automated Relation Extraction Implementation]
    "The Xunzi-Qwen1.5-14B model generates an initial dataset of character relation triplets through step-by-step reasoning, followed by dynamic scoring and filtering using the reward model. The results show that 69.79% of the data received scores above 6 (Figure 7) ... High-quality data with scores ranging from 8 to 10 were selected as the core corpus, validating the effectiveness of the fully automated knowledge extraction process and the reliability of the data filtering mechanism."

    The reward model is the filter whose reliability is being validated, and it is itself a fine-tuned Xunzi-Qwen1.5-14B—the same base model used for extraction. 'High-quality data' is defined as data receiving a reward-model score of 8-10, and then the abundance of such scores is presented as validation that the filtering mechanism is reliable. Because the score and the quality label come from the same model, the validation reduces to the filter agreeing with its own scoring; no independent human re-scoring of the selected triplets is reported. The 69.79% statistic is therefore generated by the very system under test and cannot establish the claimed reliability of the automated extraction-scoring pipeline.

full rationale

The paper's principal quantitative claims are anchored to references outside the construction loop: Table 6's F1=0.68 for Xunzi-Qwen1.5-14B is measured against the manually annotated corpus, and the headline C-CLUE comparison (DeepSeek 0.08 vs DeepSeek+RAG 0.19) uses an open-domain benchmark distinct from the Four Histories corpus. These results are not equivalent to the paper's own fitted inputs by construction. The self-citations (Xunzi model, Liu et al. 2025, Zhang et al. 2022) describe reusable artifacts and design choices rather than unverified uniqueness theorems that force the conclusions. The one genuine circularity is in Section 3.3: the reliability of the reward-model data filter is asserted from the reward model's own score distribution, so the definition of 'high-quality data' and the evidence for 'reliable filtering' come from the same model being validated. That is a self-definitional validation, though it concerns a supporting pipeline component, not the main externally benchmarked claims. The own-corpus GraphRAG result (DeepSeek 0.21 to 0.34) is evaluated on the manually annotated Four Histories dataset while the KG is populated from the same original texts; the paper never states a disjoint split, which is a test-leakage risk rather than a demonstrated circular reduction.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claims rely on hand-chosen scoring rubrics and thresholds, a borrowed ontology, and an evaluation protocol whose splits are not documented. No code or data is released, so the reader cannot audit these choices.

free parameters (3)
  • Scoring deduction points in Table 7 = 1 or 2 points per violation, total 10
    The penalty amounts define the reward model's training labels and the filter cutoff; they are chosen by hand and not justified empirically.
  • High-quality score threshold = 8 out of 10
    Section 3.3 keeps only triples with scores 8 to 10 as the core corpus; the threshold determines what enters the knowledge graph and downstream RAG, with no sensitivity analysis.
  • Fine-tuning hyperparameters = epoch=3, learning_rate=5e-5, max_length=4096, batch_size=4
    Standard values chosen without search; they affect the reported F1 but are not the paper's central contribution.
assumptions (4)
  • domain assumption The ontology model of historical text knowledge from Zhang et al. (2022) is valid and reusable for the First Four Histories.
    The paper reuses this ontology (Section 2.2.1) without validating it against an external standard; errors in the ontology propagate into extraction and retrieval.
  • ad hoc to paper The five-step chain-of-thought decomposition (translate, NER, analyze relations, build framework, extract triples) improves extraction quality.
    Section 2.3.1 asserts that this mimics human annotator cognition; the improvement is measured in-domain but the decomposition itself is a design assumption.
  • ad hoc to paper The reward model's scores correlate with ground-truth extraction quality, and using it to filter data does not bias the corpus toward the generating model's errors.
    Section 3.3 uses Xunzi-Qwen1.5-14B as both generator and scorer; the filtering loop assumes the scorer is an impartial judge rather than a mirror of the generator.
  • domain assumption The original texts from the Chinese Text Project are sufficiently accurate and representative.
    All data come from ctext.org (Section 2.2.2); no error analysis on the source is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Research on Graph-Retrieval Augmented Generation Based on Historical Text Knowledge Graphs." pith.science (2026). https://pith.science/paper/W6KKEHSG

@misc{pith2026250615241,
  author       = {Pith},
  title        = {Pith review of: Research on Graph-Retrieval Augmented Generation Based on Historical Text Knowledge Graphs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/W6KKEHSG}},
  note         = {Machine review of arXiv:2506.15241}
}
read the original abstract

This article addresses domain knowledge gaps in general large language models for historical text analysis in the context of computational humanities and AIGC technology. We propose the Graph RAG framework, combining chain-of-thought prompting, self-instruction generation, and process supervision to create a The First Four Histories character relationship dataset with minimal manual annotation. This dataset supports automated historical knowledge extraction, reducing labor costs. In the graph-augmented generation phase, we introduce a collaborative mechanism between knowledge graphs and retrieval-augmented generation, improving the alignment of general models with historical knowledge. Experiments show that the domain-specific model Xunzi-Qwen1.5-14B, with Simplified Chinese input and chain-of-thought prompting, achieves optimal performance in relation extraction (F1 = 0.68). The DeepSeek model integrated with GraphRAG improves F1 by 11% (0.08-0.19) on the open-domain C-CLUE relation extraction dataset, surpassing the F1 value of Xunzi-Qwen1.5-14B (0.12), effectively alleviating hallucinations phenomenon, and improving interpretability. This framework offers a low-resource solution for classical text knowledge extraction, advancing historical knowledge services and humanities research.

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

30 extracted references · 17 canonical work pages

  1. [1]

    Bouniot, Q. et al. (2022). (arXiv:2010.01992) 10.48550/arXiv.2010.01992

    work page Pith review arXiv doi:10.48550/arxiv.2010.01992 2022

  2. [2]

    Cheng, Z. et al. (2025). Retrieval -Augmented Generative Intelligence Question Answering Techn ology Based on Knowledge Graph. Computer Science, 52(1), pp. 87 –93. DeepSeek-A. et al. (2024). DeepSeek LLM: Scaling Open -Source Language Models with Longt ermism.http://arxiv.org/abs/2401.02954

    arXiv 2025

  3. [3]

    Dai, Z. et al. (2022). Promptagator: Few -Shot Dense Retr ieval from 8 Examples.http://arxiv.org/ abs/2209.11755

    arXiv 2022

  4. [4]

    Devlin, J. et al. (2019). BERT: Pre -Training of Deep Bidirectional Transformers for Language Understanding.http://arxiv.org/abs/1810.04805

    arXiv 2019

  5. [5]

    Gao, Y . et al. (2024). Retrieval -Augmented Generation for Lar ge Language Models: A Survey.h ttp://arxiv.org/abs/2312.10997

    arXiv 2024

  6. [6]

    He, X. et al. (2024). G -Retriever: Retrieval -Augmented Generation for Textual Graph Understan ding and Question Answering.http://arxiv.org/abs/2402.07630

    arXiv 2024

  7. [7]

    Huang, L. et al. (2025). A Survey on Hal lucination in Large Language Models: Principles, Tax onomy, Challenges, and Open Questions. ACM Transactions on Information Systems, 43 (2), pp. 1 –55. 10.1145/3703155

    doi:10.1145/3703155 2025

  8. [8]

    Kang, M. et al. (2023). Knowledge Graph -Augmented Language Models for Knowledge -Ground ed D ialogue Generation.http://arxiv.org/abs/2305.18846

    arXiv 2023

Show all 30 references
  1. [9]

    Lewis, P. et al. (2020). Retrieval -Augmented Generation for Knowledge -Intensive NLP Tasks. I n Advances in Neural Information Processing Systems. Curran Associates, Inc., pp. 9459 –74

    2020

  2. [10]

    Liang, K. (2021). Con struction and Value Analysis of the Knowledge Graph for the Special Tit le of ‘Shan Jing’. [Master’s Dissertation Thesis]. Graduate School of the Chinese Acade my of Social Sciences

    2021

  3. [11]

    Lin, C. -Y . (2004). ROUGE: A Package for Automatic Evaluation of Summaries. In Text Summ arization Branches Out. Barcelona, Spain: Association for Computational Linguistics, pp. 74–81

    2004

  4. [12]

    Liu, C. et al. (2025). Research on the Extraction and Application of Ancient Books’ Restricted Domain Relation Based on Large Language Model Technol ogy. Journal of the China So ciety for Scientific and Technical Information, 44(2), pp. 200 –19

    2025

  5. [13]

    and Cheng, kun

    Liu, J., Zhang, X. and Cheng, kun. (2023). Research on the Knowledge Service of Tibetan M edicine Ancient Books Service Platform Based on Humanities Computing. Journal of M odern Information, 43(11), pp. 47 –57

    2023

  6. [14]

    Luo, L. et al. (2024). Improve Mathematical Reasoning in Language Models by Automated Pro cess Supervision.http://arxiv.org/abs/2406.06592

    2024 arXiv

  7. [15]

    Ma, A. et al. (2022). SurveyofKnowledgeGraphBasedonReinforcementLea rning. JournalofComput erResearchandDevelopment, 59(8), pp. 1694 –722

    2022

  8. [16]

    and Feng, Y

    Ma, Y . and Feng, Y . (2024). Research on Implicit Relationship Extraction Method for Tradition al Chinese Medicine Ancient Books. Journal of Zhengzhou University(Natural Science E dition), 5 6(2), pp. 34 –42

    2024

  9. [17]

    and Hirschman, L

    Paroubek, P., Chaudiron, S. and Hirschman, L. (2007). Principles of Evaluation in Natural Lan guage Processing. Revue TAL : Traitement Automatique Des Langues, 48(1), pp. 7 –31

    2007

  10. [18]

    Peng, B. et al. (2024). Graph Retrieval -Augmented Generation: A Survey.http://arxiv.org/abs/240 8.08921

    2024

  11. [19]

    Peng, J. (2025). From Patient -Generated Health Data to Medication Recommendations: A Graph Retrieval-Augmented Generation Model for Patient Medication Information Q&A. Journa l of Modern Information, pp. 1 –22

    2025

  12. [20]

    Tang, Y . (2013). Research of Automatically Recognizing Name in Pre - Qin Ancient Chinese Classics. Modern Technology of Library and Information Service, (Z1), pp. 63 –8

    2013

  13. [21]

    Tonmoy, S. M. T. I. et al. (2024). A Comprehensive Survey of Hallucination Mitigation Techni ques in Large Language Models.http://arxiv.org/abs/2401.01313

    2024 arXiv

  14. [22]

    WANG, D. et al. (2022). Construction and Application of Pre -Trained Models of Siku Quanshu in Orientation to Digital Humanities. Library Tribune, 42(6), pp. 31 –43

    2022

  15. [23]

    Wang, S. et al. (2024). Sente nce Segmentation and Punctuation for Ancient Books Based on S upervised In -Context Training. In Sprugnoli, R. and Passarotti, M. (eds), Proceedings of the Third Workshop on Language Technologies for Historical and Ancient Languages (L T4HALA) @ L...

    2024

  16. [24]

    Wang, Y . et al. (2023). Knowledge Graph Prompting for Multi -Document Question Answering.h ttp://arxiv.org/abs/2308.11730

    2023 arXiv

  17. [25]

    Wang, Y . et al. (2023). Self -Instruct: Aligning Language Models with Self -Generated Inst ruction s.http://arxiv.org/abs/2212.10560

    2023 arXiv

  18. [26]

    Wei, J. et al. (2023). Chain -of-Thought Prompting Elicits Reasoning in Large Language Model s.http://arxiv.org/abs/2201.11903

    2023 arXiv

  19. [27]

    Yang, Z. et al. (2018). HotpotQA: A Dataset for Diverse, Explainable Multi -Hop Question Ans wering.http://arxiv.org/abs/1809.09600

    2018 arXiv

  20. [28]

    Zhang, Q. et al. (2022). Multi -Dimensional Knowledge Reorganization and Visualization of Hist ory Books: Based on Records of the Grand Historian. Journal of the China Society for Scientific and Technical Information, 41(2), pp. 130 –41

    2022

  21. [29]

    Zhang, Q. et al. (2024). Recapturing the Flow of Knowledge: Tracing Structured Knowledge B ack to Historical Records. Journal of the China Society for Scientific and Technical Info rmation, 43(4), pp. 405 –15

    2024

  22. [30]

    and Yang, F

    Zhu, L., Zhu, Y . and Yang, F. (2016). Research on Regular Expressions and Knowledge Extrac tion Based on Semantic Relationships Related to Traditional Chinese Medicine Diseases. World Science and Technology -Modernization of Traditional Chinese Medicine, 18(8), pp. 1241–50. T...

    2016

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.