Pith. sign in

REVIEW 5 major objections 5 minor 33 references

OntoRAG: Enhancing Question-Answering through Automated Ontology Derivation from Unstructured Knowledge Bases

T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read OntoRAG claims that an automatically derived ontology, built by clustering extracted entities and partitioning the resulting knowledge graph, improves LLM question-answering over unstructured technical documents, beating vector RAG in…

desk verdict A credible integration of known pieces, but the headline win rates don't survive contact with the evaluation design. read the letter →

arxiv 2506.00664 v1 pith:BN4WUGDJ submitted 2025-05-31 cs.AI cs.LG

Yash Tiwari , Owais Ahmad Lone , Mayukha Pal This is my paper
classification cs.AIcs.LG
keywords OntologyLearningRetrieval-AugmentedGenerationKnowledgeGraphsLargeLanguageModelsQuestionAnsweringCommunityDetectionSemanticWeb
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

OntoRAG is a pipeline that takes unstructured PDF documents, extracts entities and relationships, and automatically builds a queryable ontology—a structured hierarchy of classes—at several levels of granularity. The paper claims that using this ontology to retrieve context for question-answering gives large language models better global sensemaking: answers cover more aspects and more diverse perspectives. In head-to-head evaluation on electrical relay documents, OntoRAG's best level won comprehensiveness comparisons 88% of the time against vector RAG and 65% against GraphRAG's best configuration, with diversity wins of 86% and 62%. If true, this would make automated ontology creation a practical step toward query-focused summarization over technical corpora without manual knowledge engineering.

What carries the argument

The central mechanism is a two-stage clustering cascade. First, Eq. (5) groups key elements into disjoint classes by requiring cosine similarity above thresholds θname and θdef on both name and definition embeddings, using a 'K-means inspired' algorithm. Second, the resulting class-level knowledge graph is partitioned by Leiden community detection—a method that splits a graph into well-connected communities—into ontology classes, with LLM-synthesized generalized properties and inter-class relationships; repeating this on subgraphs yields the multi-level ontology O0–O3. This cascade converts flat text chunks into a hierarchical, queryable structure that supports retrieval at a chosen level of granularity.

What would settle it

Have domain experts (relay engineers), not an LLM, rate the informativeness and coverage of answers generated by OntoRAG O2 versus GraphRAG C2 on the same 125 questions. If the expert-rated comprehensiveness win rate for O2 is not significantly above chance, the reported advantage is an LLM-judge artifact. A second, complementary test: compare the automatically derived ontology classes to a manually built relay-domain ontology; if class overlap is low, the retrieval benefit cannot be attributed to faithful ontology structure.

Watch

Extended reading notes

Core claim

The paper claims that an ontology derived automatically from an unstructured corpus can serve as the retrieval index for LLM question answering, and that this beats both flat vector retrieval and GraphRAG's community-summary retrieval on global sensemaking questions. The pipeline extracts 'key elements' (entities and concepts) from text chunks, clusters them into candidate classes using cosine similarity of name and definition embeddings, builds a class-level knowledge graph, and partitions that graph with Leiden community detection into ontology classes. It then uses an LLM to synthesize class-level properties and relationships, producing a hierarchy of levels (O0–O3) via sub-community detection. The best level, O2, is reported to achieve comprehensiveness win rates of 88% against vector RAG (SS) and 65% against GraphRAG's best configuration (C2), with diversity win rates of 86% and 62%; vector RAG retains a directness edge of 92%.

Load-bearing premise

The load-bearing premise is that the clustering step—grouping key elements into classes by cosine similarity of name and definition embeddings with manually chosen thresholds—produces classes that faithfully reflect the real categories of the domain; the paper gives no direct validation of ontology quality and does not fully specify the clustering algorithm.

Editorial extensions

If this is right

  • Ontology-grounded retrieval yields higher win rates in comprehensiveness and diversity than plain vector search and GraphRAG across ontology levels O0–O3, with O2 as the reported best level.
  • Claim-based metrics indicate that answers generated through OntoRAG contain substantially more unique claims and claim clusters than vector RAG (35.2 vs 26.5 claims; 14.1 vs 8.0 clusters), suggesting denser information synthesis.
  • Because the ontology is derived automatically, the pipeline offers a path to question answering over technical corpora without manual ontology engineering, provided indexing cost is acceptable.
  • The granularity ladder (O0–O3) gives a tunable trade-off between abstraction and detail; the authors report that finer classes can improve comprehensiveness up to a point, with O2 being the sweet spot.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The 300-minute indexing time for a one-million-token corpus suggests the pipeline's practical value depends on amortizing ontology construction over many queries; a reuse-cost analysis would clarify when the added overhead pays off, which the paper does not provide.
  • Since the same LLM family generates the ontologies, the questions, and the judgments, the evaluation may partly measure self-consistency; an independent human rating of answer quality would test the generality of the 65% edge over GraphRAG.
  • A natural extension is to route queries by type: use ontology-grounded retrieval for global synthesis questions and fall back to vector search for direct local queries, which the paper's directness result hints at but does not implement.
  • The threshold-based clustering in Eq. (5) could be made adaptive by validating ontology classes against a small hand-curated seed, turning the manual threshold selection into a calibratable hyperparameter.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. OntoRAG presents a six-stage pipeline that converts unstructured PDF corpora into a hierarchical ontology via web scraping, PDF parsing, hybrid/semantic chunking, LLM-based information extraction, knowledge graph construction, Leiden community detection, and class-property synthesis. The authors evaluate OntoRAG on a proprietary electrical-relay corpus of roughly one million tokens with 125 sensemaking questions, comparing four ontology levels (O0–O3) against four GraphRAG levels (C0–C3) and a vector RAG baseline (SS), using Gemini 2.5 Flash pairwise judging and claim-based metrics. The paper reports win rates for the chosen configuration O2 of 88%/86% over SS and 65%/62% over C2 in comprehensiveness/diversity, and claims that OntoRAG improves global sensemaking while acknowledging scalability and domain-specificity limitations.

Significance. The contribution is a full pipeline that operationalizes ontology learning for an industrial domain, and the use of claim-based metrics as a second signal beyond LLM judging is a useful design. If the reported effect survived an independent evaluation, it would be a meaningful advance over vector RAG and GraphRAG for global sensemaking on technical corpora. The strengths are the explicit treatment of chunking and table extraction, the use of Leiden communities to create hierarchical ontology levels, and the candid discussion of indexing cost and prompt-domain specificity. However, the significance is currently bounded by the evaluation design: the main comparative claim rests on a same-family judge and on a post hoc selected configuration, so the empirical contribution is not yet established at the level the abstract and conclusion assert.

major comments (5)
  1. [§IV-D, §V-A] The central empirical claim is measured entirely by Gemini 2.5 Flash pairwise judgments, while Gemini 2.5 Flash is also used for entity extraction, ontology property synthesis, question generation, and answer generation (§II-D, §II-F, §III, §IV-C). This creates a same-model evaluation loop: the judge may systematically prefer outputs that match its own extraction and generation style. The paper does not report any human evaluation, a different-model judge, or a bias/self-preference check. Please provide an independent blinded evaluation (human raters or at least a different LLM family with agreement analysis) before the superiority claim can be accepted.
  2. [Abstract and §V-A/§VII] The abstract reports comprehensiveness win rates of 85% against vector RAG and 75% against GraphRAG, whereas §V-A and §VII report 88% and 65%. The paper never explains which numbers are correct; this is a verification blocker for the main result. Additionally, the choice of O2 as the 'best' configuration is made after inspecting all levels, and C2 is likewise selected post hoc; the reported p-values from Wilcoxon tests do not account for this selection, so they overstate significance. Please reconcile the numbers and either pre-register the configuration selection or correct for multiple comparisons across all nine conditions.
  3. [§II-E, Eq. (5)] The clustering step that defines ontology classes is under-specified: the paper calls it a 'K-means inspired algorithm [33]' but gives no pseudocode, no batching or merging procedure, no choice of embeddings, and no sensitivity analysis for thresholds θname and θdef. Reference [33] is also incomplete. More importantly, there is no direct validation that the derived classes correspond to meaningful domain categories. Please specify the algorithm completely, validate the ontology against human-annotated classes or a gold taxonomy, and report how results vary with the thresholds and the number of Leiden communities k.
  4. [Table II, §V-C] The claim-based metrics show O2 with 35.2 claims and 14.1 clusters versus C2 with 34.8 and 13.8—differences of 0.4 and 0.3 units—and no error bars or significance tests are reported. The text in §V-C calls this 'a slight but significant improvement,' but no evidence supports significance at this granularity. Please report per-replicate variance, confidence intervals, and appropriate tests, or soften the claim to a difference that is not statistically established.
  5. [§I, GraphRAG characterization] The Introduction states that GraphRAG 'typically assumes the availability of a pre-existing ontology,' but the cited GraphRAG paper [8] builds a knowledge graph from extracted entities and relationships and does not require an ontology. Please correct this characterization, since part of the paper's motivation rests on it.
minor comments (5)
  1. [§II-D, §II-F] The text repeatedly says 'See Appendix' for prompts, table extraction, atomic fact extraction, and JSON schemas, but no appendix is present in the full text; please include the appendix or a supplementary file.
  2. [Fig. 4 caption] The caption says the curves are 'averaged over comparisons with SS,' but the x-axis and legend compare OntoRAG and GraphRAG levels; please clarify what is being averaged and whether the line plot reflects direct pairwise comparisons between the two systems.
  3. [Abstract] In the abstract, 'OntoRAGs effectiveness' should be 'OntoRAG's effectiveness.'
  4. [Table I caption] Table I reports p-values only as inequalities; please give the exact p-values or state the corrected significance threshold so readers can verify the Holm-Bonferroni correction.
  5. [Reference [33]] Reference [33] is incomplete: it lacks author names, a title, and full bibliographic data, showing only a DOI URL; please complete it.

Circularity Check

0 steps flagged · score 0.0 of 10

No derivation in OntoRAG reduces to its own inputs; the reported win rates are empirical measurements, not predictions derived by construction, so no circularity is established.

full rationale

OntoRAG's pipeline (PDF parsing, chunking, extraction, clustering, graph construction, community detection, retrieval, answer generation) is an empirical system, and the headline win rates in Section V-A are observed outcomes of pairwise comparisons, not quantities derived by construction from fitted parameters. No equation in the paper equates a predicted result to an input: Eq. (5) defines clustering via cosine similarity and thresholds, but the comprehensiveness/diversity win rates are not algebraically or statistically forced by that clustering definition. The use of Gemini 2.5 Flash for extraction, question generation, answer generation, and judging is a legitimate methodological validity concern (LLM-as-judge self-preference risk), and the abstract/body win-rate discrepancy (85%/75% vs 88%/65%) plus the under-specified 'K-means inspired algorithm' referenced as [33] are rigor and reproducibility issues, but none of these constitutes circularity in the sense of a claim reducing to its own premises by construction. Section VI's stated limitations on scalability and domain-specific prompts are acknowledged by the paper and do not reveal a circular derivation. There are no load-bearing self-citations. The central empirical claim therefore has independent content, even though its experimental support is open to methodological critique.

Assumptions & free parameters 9 free parameters · 5 assumptions · 0 invented entities

The central claim depends on a chain of unverified or hand-tuned choices: embedding-similarity thresholds that define ontology classes, manually selected chunking parameters, a human-chosen community count, and a post hoc chosen retrieval level. The evaluation then leans on the same LLM family for both construction and judging, so these parameters are not validated against an external gold standard.

free parameters (9)
  • theta_name similarity threshold (θname) = not reported
    Controls the formation of ontology classes in Eq. (5); the ontology structure and all downstream answers depend on it.
  • theta_def similarity threshold (θdef) = not reported
    Applied to definition embeddings in Eq. (5); together with θname it determines which key elements are clustered into the same class.
  • Hybrid chunking minimum length threshold = not reported
    Determines chunk boundaries in Section II-C; affects what context is retrieved and passed to the LLM.
  • Semantic chunking similarity threshold = selected by manual supervision
    Controls merging of adjacent sentences in Section II-C2; chosen by inspecting chunk sizes, not by a principled rule.
  • Number of ontology classes k (Leiden communities) = not reported, chosen by human-in-the-loop
    Guides ontology granularity in Section II-F1; the paper reports O0-O3 levels, and O2 was selected after seeing the results.
  • Retrieval ontology level = O2
    The best-performing level is selected post hoc from the same evaluation used to report the headline win rates.
  • Context expansion window = 200 tokens
    Configurable parameter in Section III that changes the retrieved context length around each matched chunk.
  • PDF padding values = 100 vertical, 20 horizontal units
    Manual choices in Eq. (3)-(4) to capture contextual elements around table regions.
  • Chunk size and overlap = 600 tokens per chunk, 100-token overlap
    Chosen for the corpus in Section IV-A; affects the granularity of retrieval units.
assumptions (5)
  • domain assumption Gemini 2.5 Flash produces faithful entities and relationships during information extraction.
    All downstream graph and ontology content is built from these extractions; no human verification is reported.
  • domain assumption Cosine similarity of name and definition embeddings is a valid signal for grouping entities into ontology classes.
    The central clustering step in Eq. (5) assumes semantic proximity of embeddings corresponds to ontological category membership.
  • domain assumption The LLM judge (Gemini 2.5 Flash) provides unbiased pairwise quality scores.
    Win rates in Table I are produced by the same model family used to build the ontology and generate answers; no independent judge or human gold standard is used.
  • standard math Leiden community detection produces meaningful ontology classes from the candidate-class graph.
    Leiden is a standard graph algorithm, but its use as a proxy for ontology construction is an assumption about the graph structure.
  • domain assumption The proprietary 1M-token relay corpus is representative enough to support general claims of superiority.
    Single-domain, single-corpus evaluation with no public dataset or cross-domain validation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of OntoRAG: Enhancing Question-Answering through Automated Ontology Derivation from Unstructured Knowledge Bases." pith.science (2026). https://pith.science/paper/BN4WUGDJ

@misc{pith2026250600664,
  author       = {Pith},
  title        = {Pith review of: OntoRAG: Enhancing Question-Answering through Automated Ontology Derivation from Unstructured Knowledge Bases},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BN4WUGDJ}},
  note         = {Machine review of arXiv:2506.00664}
}
read the original abstract

Ontologies are pivotal for structuring knowledge bases to enhance question answering (QA) systems powered by Large Language Models (LLMs). However, traditional ontology creation relies on manual efforts by domain experts, a process that is time intensive, error prone, and impractical for large, dynamic knowledge domains. This paper introduces OntoRAG, an automated pipeline designed to derive ontologies from unstructured knowledge bases, with a focus on electrical relay documents. OntoRAG integrates advanced techniques, including web scraping, PDF parsing, hybrid chunking, information extraction, knowledge graph construction, and ontology creation, to transform unstructured data into a queryable ontology. By leveraging LLMs and graph based methods, OntoRAG enhances global sensemaking capabilities, outperforming conventional Retrieval Augmented Generation (RAG) and GraphRAG approaches in comprehensiveness and diversity. Experimental results demonstrate OntoRAGs effectiveness, achieving a comprehensiveness win rate of 85% against vector RAG and 75% against GraphRAGs best configuration. This work addresses the critical challenge of automating ontology creation, advancing the vision of the semantic web.

Figures

Figures reproduced from arXiv: 2506.00664 by the authors.

Figure 1
Figure 1. Overview of the Retrieval-Augmented Generation (RAG) process for question-answering, consisting of three main stages: Indexing, Retrieval, and [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Workflow of the OntoRAG pipeline, illustrating the transformation of unstructured PDF documents into a structured ontology through sequential [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Detailed workflow of the OntoRAG pipeline, with emphasis on the Ontology Creation stage. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Line plot comparing comprehensiveness and diversity win rates of OntoRAG (O0–O3) and GraphRAG (C0–C3) across their respective levels, averaged [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

33 extracted references · 25 canonical work pages

  1. [33]

    Ikotun, Absalom E

    Abiodun M. Ikotun, Absalom E. Ezugwu, Laith Abualigah, Belal Abuhaija, Jia Heming,. https://doi.org/10.1016/j.ins.2022.11.139

    doi:10.1016/j.ins.2022.11.139 2022

  2. [8]

    From local to global: A Graph RAG approach to query- focused summarization,

    D. Edge et al., “From local to global: A Graph RAG approach to query- focused summarization,” arXiv preprint arXiv:2404.16130 , 2024

    arXiv 2024

  3. [1]

    Language models are few-shot learners,

    T. Brown et al., “Language models are few-shot learners,” in Advances in Neural Information Processing Systems , vol. 33, 2020, pp. 1877–1901

    work page 2020

  4. [2]

    On the opportunities and risks of foundation models,

    R. Bommasani et al., “On the opportunities and risks of foundation models,” arXiv preprint arXiv:2108.07258 , 2021

    arXiv 2021

  5. [3]

    PaLM: Scaling language modeling with path- ways,

    A. Chowdhery et al., “PaLM: Scaling language modeling with path- ways,” Journal of Machine Learning Research , vol. 24, no. 240, pp. 1–113, 2023

    work page 2023

  6. [4]

    Large language models struggle to learn long-tail knowledge,

    N. Kandpal et al., “Large language models struggle to learn long-tail knowledge,” in International Conference on Machine Learning , 2023, pp. 15,696–15,707

    work page 2023

  7. [5]

    Siren’s song in the AI ocean: A survey on hallucination in large language models,

    Y . Zhang et al., “Siren’s song in the AI ocean: A survey on hallucination in large language models,” arXiv preprint arXiv:2309.01219 , 2023

    arXiv 2023

  8. [6]

    Nearest neighbor machine translation,

    U. Khandelwal et al., “Nearest neighbor machine translation,” in Inter- national Conference on Learning Representations , 2021

    work page 2021

Show all 33 references
  1. [7]

    DUC 2005: Evaluation of question-focused summarization systems,

    H. T. Dang, “DUC 2005: Evaluation of question-focused summarization systems,” in Proceedings of the Workshop on Task-Focused Summariza- tion and Question Answering , 2006, pp. 48–55

    2005

  2. [9]

    Knowledge repository of ontology learning tools from text,

    A. Konys, “Knowledge repository of ontology learning tools from text,” Procedia Computer Science , vol. 159, pp. 1614–1628, 2019

    2019

  3. [10]

    The unified medical language system (UMLS): Inte- grating biomedical terminology,

    O. Bodenreider, “The unified medical language system (UMLS): Inte- grating biomedical terminology,” Nucleic Acids Research , vol. 32, no. suppl 1, pp. D267–D270, 2004

    2004

  4. [11]

    WordNet: A lexical database for English,

    G. A. Miller, “WordNet: A lexical database for English,” Communica- tions of the ACM , vol. 38, no. 11, pp. 39–41, 1995

    1995

  5. [12]

    Y AGO: A multilingual knowledge base from Wikipedia, WordNet, and GeoNames,

    T. Rebele et al., “Y AGO: A multilingual knowledge base from Wikipedia, WordNet, and GeoNames,” in The Semantic Web–ISWC 2016, 2016, pp. 177–185

    2016

  6. [13]

    The Dublin Core Metadata Initiative,

    S. L. Weibel and T. Koch, “The Dublin Core Metadata Initiative,” D-Lib Magazine, vol. 6, no. 12, 2000

    2000

  7. [14]

    Schema.org: Evolution of structured data on the web,

    R. V . Guha et al., “Schema.org: Evolution of structured data on the web,” Communications of the ACM , vol. 59, no. 2, pp. 44–51, 2016

    2016

  8. [15]

    Ontology development 101: A guide to creating your first ontology,

    N. F. Noy and D. L. McGuinness, “Ontology development 101: A guide to creating your first ontology,” Stanford Knowledge Systems Laboratory Technical Report KSL-01-05, 2001

    2001

  9. [16]

    Methodologies for ontology development,

    D. Jones, T. Bench-Capon, and P. Visser, “Methodologies for ontology development,” University of Liverpool, 1998

    1998

  10. [17]

    Towards learn- ing from user feedback for ontology-based information extraction,

    K. Opasjumruskit, S. Schindler, and P. M. Schafer, “Towards learn- ing from user feedback for ontology-based information extraction,” in Proceedings of the 1st International Workshop on Challenges and Experiences from Data Integration to Knowledge Graphs , 2019

    2019

  11. [18]

    Multi-hop question answering over knowledge graphs using large language models,

    A. Chakraborty, “Multi-hop question answering over knowledge graphs using large language models,” arXiv preprint arXiv:2404.19234 , 2024

    2024 arXiv

  12. [19]

    Multi-hop question answering under temporal knowl- edge editing,

    K. Cheng et al., “Multi-hop question answering under temporal knowl- edge editing,” arXiv preprint arXiv:2404.00492 , 2024

    2024 arXiv

  13. [20]

    FanOutQA: A multi-hop, multi-document question answering benchmark for large language models,

    A. Zhu et al., “FanOutQA: A multi-hop, multi-document question answering benchmark for large language models,” arXiv preprint arXiv:2402.14116, 2024

    2024 arXiv

  14. [21]

    From human experts to machines: An LLM supported approach to ontology and knowledge graph construction,

    V . K. Kommineni, B. K ¨onig-Ries, and S. Samuel, “From human experts to machines: An LLM supported approach to ontology and knowledge graph construction,” arXiv preprint arXiv:2403.08345 , 2024

    2024 arXiv

  15. [22]

    Berners-Lee, Weaving the Web: The Original Design and Ultimate Destiny of the World Wide Web

    T. Berners-Lee, Weaving the Web: The Original Design and Ultimate Destiny of the World Wide Web . HarperCollins, 2000

    2000

  16. [23]

    BeautifulSoup,

    “BeautifulSoup,” [Online]. Available: https://www.crummy.com/software/BeautifulSoup/

  17. [24]

    Available: https://scrapy.org/

    “Scrapy,” [Online]. Available: https://scrapy.org/

  18. [25]

    Unstructured Library,

    “Unstructured Library,” [Online]. Available: https://unstructured.io/

  19. [26]

    PDFMiner,

    “PDFMiner,” [Online]. Available: https://github.com/pdfminer/pdfminer.six

  20. [27]

    PyMuPDF,

    “PyMuPDF,” [Online]. Available: https://pymupdf.readthedocs.io/

  21. [28]

    Amazon Textract,

    “Amazon Textract,” [Online]. Available: https://aws.amazon.com/textract/

  22. [29]

    Pytesseract,

    “Pytesseract,” [Online]. Available: https://github.com/madmaze/pytesseract

  23. [30]

    From Louvain to Leiden: Guaranteeing well-connected communities,

    V . A. Traag, L. Waltman, and N. J. van Eck, “From Louvain to Leiden: Guaranteeing well-connected communities,” Scientific Reports , vol. 9, no. 1, pp. 1–12, 2019

    2019

  24. [31]

    graspologic,

    “graspologic,” [Online]. Available: https://microsoft.github.io/graspologic/

  25. [32]

    Scikit-learn,

    “Scikit-learn,” [Online]. Available: https://scikit-learn.org/

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.