REVIEW 4 major objections 6 minor 35 references
PolyUQuest: Verifiable Structure-Aware Web RAG over Heterogeneous Graphs
T0 review · 4 major / 6 minor · reviewed 2026-07-10 · grok-4.5
Pith's one-line read A structure-aware web RAG system that unifies hyperlinks, page hierarchy, and entities, then routes each query to a matching retrieval mode, answers more correctly and faithfully while using far fewer tokens than prior systems.
desk verdict Solid CIKM-style systems demo: unifies hyperlink/DOM/entity structure with mode routing and real provenance, beats relevant baselines on a PolyU crawl with clear token savings; main soft spot is the author-built 300-question set aligned to those modes. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The three-layer heterogeneous web graph—webpage nodes, heading-aware evidence-block nodes, and entity/topic nodes—plus a two-tier router that selects among direct block retrieval, cross-page navigation, and multi-hop entity reasoning. The graph supports moving from an entity to its blocks, containing page, and linked neighbors; the router avoids the large global contexts that make other graph RAG systems expensive.
What would settle it
Build a new multi-type question set over the same or a comparable institutional website without reference to the three-mode taxonomy, re-run all systems with identical generators and embeddings, and check whether the correctness, coverage, and faithfulness gaps remain.
Extended reading notes
Core claim
Jointly encoding hyperlink topology, DOM hierarchy, and cross-page entity relations in a single heterogeneous graph, then dispatching each query to a structure-matched retrieval mode, produces more correct, complete, and evidence-faithful answers than systems that flatten pages or inject large global graph contexts, while consuming substantially fewer language-model tokens per query.
Load-bearing premise
The evaluation questions were written to cover the three retrieval modes the system implements, with manual labels of relevant pages and blocks; if that design favors the system’s own structure, the reported gains over baselines would shrink.
Editorial extensions
If this is right
- Organizational websites can support multi-page QA with higher faithfulness without the token cost of community-summary graph RAG.
- Citations that include source page, heading path, and entity links let users verify claims without opening and comparing many pages.
- Heading-aware DOM blocks, not fixed-size chunks, are the main quality driver according to the ablations.
- Routing by structural need keeps average query cost near simple retrieval while improving cross-page and multi-hop answers.
- Porting to a new domain needs a fresh crawl and entity schema; indexing, routing, and provenance stay the same.
Reading between the lines
- Independent question sets not designed around a three-mode taxonomy will be needed to confirm that gains transfer beyond systems that share that taxonomy.
- Provenance-carrying block units may reduce the need for separate fact-checking layers in production institutional chatbots.
- The same three-layer index could help decide when an agent should follow a hyperlink versus expand entity relations already stored in the graph.
- Token reductions of this scale could make structure-aware RAG practical for mid-size sites that cannot afford full offline community-summary pipelines.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. PolyUQuest is a structure-aware web RAG system that models a website as a heterogeneous graph unifying hyperlink topology (pages), DOM hierarchy (heading-aware evidence blocks), and cross-page entity–relation knowledge. A two-tier router (heuristics then LLM) dispatches each query to one of three modes: Mode A direct block retrieval, Mode B cross-page navigation after query decomposition, and Mode C multi-hop entity/topic reasoning with a three-term block score (Eq. 1). Answers are generated with block-level provenance (source page, heading path, entity links). On a 4,240-page PolyU crawl and an author-built 300-question PolyU-Web set, the system reports higher correctness, coverage, and faithfulness than ChunkRAG, HtmlRAG, FastGraphRAG, and LightRAG at substantially lower query-token cost (Table 1), with ablations attributing most quality loss to removing DOM blocks (Table 2). A demo interface supports citation inspection, retrieval traces, and graph exploration; deployment as a student QA service is planned.
Significance. If the comparative gains hold under less system-aligned evaluation, the work is a useful systems contribution for organizational websites: it jointly uses three structural layers that prior RAG lines usually treat separately, pairs them with intent-matched retrieval rather than global graph context, and makes every claim inspectable via block provenance. The reported ~10× query-token reduction versus LightRAG and the high faithfulness (0.921) are practically meaningful for deployment. Strengths that should be credited include a concrete three-layer index over a real multi-thousand-page crawl, open demo/code, explicit mode routing with a stated scoring formula, and ablations that isolate DOM segmentation as the main quality driver. The main significance risk is that superiority is currently demonstrated only on a single-domain, author-constructed benchmark whose taxonomy mirrors the three modes.
major comments (4)
- [Performance Highlights / Table 1] Performance Highlights / Table 1: The central superiority claim (Corr. 0.644, Cov. 0.649, Faith. 0.921, Q.Tok. 2,968) rests on 300 author-constructed PolyU-Web questions that are explicitly designed to cover the three retrieval needs Modes A/B/C implement, with manual gold pages and blocks. That alignment risks rewarding the paper’s own routing taxonomy and block granularity rather than measuring general structure-aware retrieval. Please report per-mode (or per-question-type) metrics for all systems, describe how questions and gold blocks were sampled/annotated to avoid mode favoritism, and ideally add an independently written or mode-agnostic hold-out set (or external organizational site). Without this, the 36-point faithfulness gap and token advantage over LightRAG remain hard to interpret as general.
- [Table 2 / §2.2 Mode C] Table 2 only ablates “w/o DOM blocks” and “w/o cross-page mode.” There is no ablation of the entity graph / Mode C path, of the two-tier router (e.g., always Mode A), or of entity resolution quality, even though Mode C and the heterogeneous entity layer are core claimed contributions. A Mode-C-off or entity-edges-off variant on the multi-hop subset is needed to show that the entity layer, not only DOM blocks, drives gains on the questions it is meant to serve.
- [Table 1] Table 1 reports point estimates with no variance, confidence intervals, or significance tests over the 300 questions (or over multiple generator seeds). Given small absolute margins versus LightRAG on Corr./Cov. (0.644 vs 0.610; 0.649 vs 0.612), statistical support is load-bearing for the “outperforms existing RAG systems” claim. Please add error bars (e.g., bootstrap over questions) and, if LLM-as-judge or human scoring is used for Corr./Cov./Faith., the protocol and inter-annotator agreement.
- [§2.2 Two-Tier Router] §2.2 Two-Tier Router: Routing accuracy is not measured. Heuristic triggers (“which professors” → C, “admission requirements for” → B) and the second-tier LLM classifier with confidence are load-bearing for both quality and the low token budget, yet misroutes would systematically hurt Modes B/C questions. Report router accuracy (and confusion among A/B/C) on the benchmark, and the fraction of queries handled by heuristics vs LLM, so readers can separate routing skill from retrieval skill.
minor comments (6)
- [§2.2 Eq. (1)] Eq. (1): State how κ=30 and the same-page penalty weight were chosen (validation procedure, sensitivity). Free parameters (block word threshold 150, ANN/BM25 cutoffs, rerank depths in Fig. 2) should be listed in one place for reproducibility.
- [§2.1 Entity Extraction / Generalizability] Generalizability Discussion correctly notes the need for a domain-specific entity schema, but the paper never reports entity-type inventory, extraction prompt, or resolution precision/recall. A short quantitative note on entity quality would strengthen Layer 3 claims.
- [Table 1 / Performance Highlights] Baselines: Confirm whether HtmlRAG and ChunkRAG received the same multi-page crawl and hyperlink access (or only per-page HTML/text). If baselines cannot follow links, Mode B/C gains partly reflect capability mismatch rather than ranking quality; state this limitation explicitly next to Table 1.
- [Figure 2] Figure 2 is dense; a clearer legend for Mode A/B/C trace panels and consistent citation numbering in the chat example would help demo readers.
- [§2.3 / Mode C] Minor wording: “Structure-A ware Retrieval” (hyphen/space) in the Online Query Pipeline list; “multi-pages” → “multiple pages” in Mode C; arXiv ID/date in the banner (2607.08269 / Jul 2026) should be checked for consistency with the submission venue line (CIKM ’26).
- [Table 1] Build tokens (17.5M) are lower than LightRAG’s 37.4M; briefly say whether entity extraction LLM calls dominate offline cost and whether the same LLM family is used offline and online.
Circularity Check
No derivation circularity: empirical systems paper with self-contained graph/routing design and external baselines; custom benchmark alignment is evaluation bias risk, not Eq.=input by construction.
full rationale
PolyUQuest is an engineering/systems contribution: offline heterogeneous graph construction (pages, DOM blocks, entities), a two-tier router into Modes A/B/C, and empirical comparison on PolyU-Web. There is no claimed first-principles derivation whose conclusion reduces to its premises by definition. Equation (1) is an explicit engineered ranking score (cosine + capped entity coverage + length), not a fitted quantity re-labeled as a prediction. Hyperparameters (150-word blocks following prior HTML/Graph RAG work; κ=30 tuned on validation) are stated as design choices, not as forecasts of held-out identities. Central results (Table 1 correctness/coverage/faithfulness and token counts) are measured against external baselines (ChunkRAG, HtmlRAG, FastGraphRAG, LightRAG) under a shared generator/embedding setup; ablations (Table 2) remove components rather than tautologically recover the full system. Self-citations among coauthors appear in related RAG work but are not load-bearing uniqueness theorems that force the architecture. The author-built 300-question set deliberately covering the three retrieval modes is a selection/fairness concern for generalization, not circularity under the required patterns (no self-definitional identity, no fitted-input-as-prediction, no uniqueness imported from authors). Score 0 with empty steps is the proportionate finding.
Assumptions & free parameters
free parameters (3)
- block_word_threshold =
150 words
- entity_coverage_cap_kappa =
30
- retrieval_and_rerank_cutoffs =
top-50 then top-10 (demo traces)
assumptions (4)
- domain assumption Website knowledge for institutional QA is adequately captured by hyperlink topology + heading-aware DOM blocks + extracted entity–relation graphs.
- domain assumption A two-tier router (heuristics then LLM classifier) can assign queries to one of three structural modes with enough accuracy that mode-specific retrieval improves end quality.
- domain assumption LLM-based entity/relation extraction plus alias resolution yields a graph faithful enough for multi-hop Mode C reasoning.
- standard math Standard dense+BM25 retrieval, cross-encoder reranking, and LLM answer generation are valid building blocks whose relative gains can be attributed to structure and routing.
invented entities (2)
-
Three-layer heterogeneous web graph (page, block, entity, topic nodes)
-
Two-tier structure-driven query router with Modes A/B/C
Cite this review
Pith. "Pith review of PolyUQuest: Verifiable Structure-Aware Web RAG over Heterogeneous Graphs." pith.science (2026). https://pith.science/paper/UY5B7GON
@misc{pith2026260708269,
author = {Pith},
title = {Pith review of: PolyUQuest: Verifiable Structure-Aware Web RAG over Heterogeneous Graphs},
year = {2026},
howpublished = {\url{https://pith.science/paper/UY5B7GON}},
note = {Machine review of arXiv:2607.08269}
}
read the original abstract
Existing retrieval-augmented generation (RAG) systems treat web pages as flat text, losing the structural and semantic signals encoded in HTML. We present PolyUQuest, a verifiable, structure-aware web RAG framework built on a heterogeneous graph that unifies hyperlink topology between pages, DOM hierarchy within pages, and entity-relation knowledge across pages. A two-tier router dispatches each query to one of three retrieval modes matched to its structural need, including direct block retrieval, cross-page graph traversal, and multi-hop entity reasoning. Every answer is fully verifiable, as each cited block carries its source page, heading path, and entity links so that users can trace any claim back to its structural evidence. We evaluate on the official websites of the Hong Kong Polytechnic University (PolyU), comprising 4,240 pages, 31,086 DOM blocks, 29,119 entities, and 37,680 relations, together with a multi-type evaluation benchmark. PolyUQuest outperforms existing RAG systems in answer correctness, coverage, and faithfulness, while consuming significantly fewer LLM tokens per query. The demonstration provides an interactive interface for inspecting cited answers, comparing retrieval traces across routing modes, and exploring evidence graph paths. PolyUQuest is being prepared for deployment as a student-facing QA service at PolyU.
Figures
Reference graph
Works this paper leans on
-
[1]
Qi Chen, Xiubo Geng, Corby Rosset, Carolyn Buractaon, Jingwen Lu, Tao Shen, Kun Zhou, Chenyan Xiong, Yeyun Gong, Paul Bennett, et al. 2024. Ms marco web search: A large-scale information-rich web dataset with millions of real click labels. InCompanion Proceedings of the ACM Web Conference 2024. 292–301
work page 2024
-
[2]
Nayoung Choi, Grace Byun, Andrew Chung, Ellie S Paek, Shinsun Lee, and Jinho D Choi. 2025. Reference-Aligned Retrieval-Augmented Question Answering over Heterogeneous Proprietary Documents. InProceedings of the 34th ACM International Conference on Information and Knowledge Management. 5626–5633
work page 2025
-
[3]
Circlemind. 2024. Fast GraphRAG. https://github.com/circlemind-ai/fast- graphrag
work page 2024
-
[4]
Wenqi Fan, Pangjing Wu, Yujuan Ding, Liangbo Ning, Shijie Wang, and Qing Li
-
[5]
In2025 IEEE 41st International Conference on Data Engineering (ICDE)
Towards Retrieval-Augmented Large Language Models: Data Management and System Design. In2025 IEEE 41st International Conference on Data Engineering (ICDE). IEEE, 4509–4512
-
[6]
Zirui Guo, Lianghao Xia, Yanhua Yu, Tian Ao, and Chao Huang. 2024. Lightrag: Simple and fast retrieval-augmented generation.arXiv preprint arXiv:2410.05779 2, 3 (2024)
work page Pith review arXiv 2024
-
[7]
Chen Han, Yuanyuan Li, and Xijin Tang. 2025. DocPolicyKG: A Lightweight LLM-Based Framework for Knowledge Graph Construction from Chinese Policy Documents. InProceedings of the 34th ACM International Conference on Informa- tion and Knowledge Management. 4753–4757
work page 2025
-
[8]
Farnoosh Hashemi and Laks VS Lakshmanan. 2025. KRAFT: A Knowledge Graph- Based Framework for Automated Map Conflation. InProceedings of the 34th ACM International Conference on Information and Knowledge Management. 802–812
work page 2025
Show all 35 references
-
[9]
Yuntong Hu, Zhihan Lei, Zheng Zhang, Bo Pan, Chen Ling, and Liang Zhao. 2025. Grag: Graph retrieval-augmented generation. InFindings of the Association for Computational Linguistics: NAACL 2025. 4145–4157
2025
-
[10]
Peng Huang, Meihui Zhang, Ziyue Zhong, Chengliang Chai, and Ju Fan. 2024. Representation learning for entity alignment in knowledge graph: A design space exploration. In2024 IEEE 40th International Conference on Data Engineering (ICDE). IEEE, 3462–3475
2024
-
[11]
Yiqian Huang, Shiqi Zhang, and Xiaokui Xiao. 2025. Ket-rag: A cost-efficient multi-granular indexing framework for graph-rag. InProceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 2. 1003–1012
2025
-
[12]
Pengcheng Jiang, Siru Ouyang, Yizhu Jiao, Ming Zhong, Runchu Tian, and Jiawei Han. 2025. Retrieval and structuring augmented generation with large language models. InProceedings of the 31st ACM SIGKDD Conference on Knowledge Discov- ery and Data Mining V. 2. 6032–6042
2025
-
[13]
Wenqi Jiang, Shuai Zhang, Boran Han, Jie Wang, Bernie Wang, and Tim Kraska
-
[14]
InProceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V
Piperag: Fast retrieval-augmented generation via adaptive pipeline paral- lelism. InProceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 1. 589–600
-
[15]
Da Li, Keping Bi, Jiafeng Guo, and Xueqi Cheng. 2025. Bridging Queries and Tables through Entities in Open-Domain Table Retrieval. InProceedings of the 34th ACM International Conference on Information and Knowledge Management. 1540–1550
2025
-
[16]
Mingyang Li, Shisong Chen, Shengkun Tu, Ziyi Du, Jinghao Zhang, Zhixu Li, and Yanghua Xiao. 2025. KUG: Joint Enhancement of Internal and External Knowledge for Retrieval-Augmented Generation. InProceedings of the 34th ACM International Conference on Information and Knowledge M...
2025
-
[17]
Peizheng Li, Chaoyi Chen, Hao Yuan, Zhenbo Fu, Hang Shen, Xinbo Yang, Qiange Wang, Xin Ai, Yanfeng Zhang, Yingyou Wen, et al. 2025. NeutronRAG: Towards Understanding the Effectiveness of RAG from a Data Retrieval Perspective. In Companion of the 2025 International Conference o...
2025
-
[18]
Rui Li, Quanyu Dai, Zeyu Zhang, Xu Chen, Zhenhua Dong, and Ji-Rong Wen
-
[19]
InProceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V
Knowtrace: Bootstrapping iterative retrieval-augmented generation with structured knowledge tracing. InProceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 2. 1470–1480
-
[20]
Lei Liang, Zhongpu Bo, Zhengke Gui, Zhongshu Zhu, Ling Zhong, Peilong Zhao, Mengshu Sun, Zhiqiang Zhang, Jun Zhou, Wenguang Chen, et al . 2025. Kag: Boosting llms in professional domains via knowledge augmented generation. In Companion Proceedings of the ACM on Web Conference ...
2025
-
[21]
Yiming Lin, Madelon Hulsebos, Ruiying Ma, Shreya Shankar, Sepanta Zeighami, Aditya G Parameswaran, and Eugene Wu. 2025. Querying templatized document collections with large language models. In2025 IEEE 41st International Conference on Data Engineering (ICDE). IEEE, 2422–2435
2025
-
[22]
Boci Peng, Yun Zhu, Yongchao Liu, Xiaohe Bo, Haizhou Shi, Chuntao Hong, Yan Zhang, and Siliang Tang. 2025. Graph retrieval-augmented generation: A survey. ACM Transactions on Information Systems44, 2 (2025), 1–52
2025
-
[23]
Xingrun Quan, Yongkang Zhou, and Junjie Yao. 2025. ThoughtForest-KGQA: A Multi-Chain Tree Search for Knowledge Graph Reasoning. InProceedings of the 34th ACM International Conference on Information and Knowledge Management. 5156–5160
2025
-
[24]
Ishneet Sukhvinder Singh, Ritvik Aggarwal, Ibrahim Allahverdiyev, Muhammad Taha, Aslihan Akalin, Kevin Zhu, and Sean O’Brien. 2024. Chunkrag: Novel llm-chunk filtering method for rag systems.arXiv preprint arXiv:2410.19572 (2024)
2024 arXiv
-
[25]
Jiejun Tan, Zhicheng Dou, Wen Wang, Mang Wang, Weipeng Chen, and Ji- Rong Wen. 2025. Htmlrag: Html is better than plain text for modeling retrieved knowledge in rag systems. InProceedings of the ACM on Web Conference 2025. 1733–1746
2025
-
[26]
Yubo Wang, Haoyang Li, Fei Teng, and Lei Chen. 2025. AGRAG: Advanced Graph- based Retrieval-Augmented Generation for LLMs.arXiv preprint arXiv:2511.05549 (2025)
2025
-
[27]
Yubo Wang, Haoyang Li, Fei Teng, and Lei Chen. 2026. GORAG: Graph-based Online Retrieval Augmented Generation for Dynamic Few-shot Social Media Text Classification. InProceedings of the ACM Web Conference 2026(United Arab Emirates)(WWW ’26). Association for Computing Machinery...
2026 doi
-
[28]
Yu Wang, Nedim Lipka, Ruiyi Zhang, Alexa Siu, Yuying Zhao, Bo Ni, Xin Wang, Ryan Rossi, and Tyler Derr. 2024. Topology-aware retrieval augmentation for text generation. InProceedings of the 33rd ACM International Conference on Information and Knowledge Management. 2442–2452
2024
-
[29]
Zhishang Xiang, Chuanjie Wu, Qinggang Zhang, Shengyuan Chen, Zijin Hong, Xiao Huang, and Jinsong Su. 2025. When to use graphs in rag: A compre- hensive analysis for graph retrieval-augmented generation.arXiv preprint arXiv:2506.05690(2025)
2025
-
[30]
Yuming Xu, Mingtao Zhang, Zhuohan Ge, Haoyang Li, Nicole Hu, Yongqi Zhang, Zhiyuan Wen, Jason Chen Zhang, Qing Li, and Lei Chen. 2026. Securing Retrieval- Augmented Generation: A Taxonomy of Attacks, Defenses, and Future Directions. arXiv:2604.08304 [cs.CR] https://arxiv.org/a...
2026 arXiv
-
[31]
Dezhi Ye, Ye Qin, Bowen Tian, Jiabin Fan, Jie Liu, Haijin Liang, and Jin Ma. 2025. Can LLMs Really Help Query Understanding In Web Search? A Practical Per- spective. InProceedings of the 34th ACM International Conference on Information and Knowledge Management. 5433–5438
2025
-
[32]
Runjie Yu, Weizhou Huang, Shuhan Bai, Jian Zhou, and Fei Wu. 2025. AquaPipe: A Quality-Aware Pipeline for Knowledge Retrieval and Large Language Models. Proceedings of the ACM on Management of Data3, 1 (2025), 1–26
2025
-
[33]
Shuo Yu, Mingyue Cheng, Qi Liu, Daoyu Wang, Jiqian Yang, Jie Ouyang, Yucong Luo, Chenyi Lei, and Enhong Chen. 2025. Multi-source knowledge pruning for retrieval-augmented generation: A benchmark and empirical study. InProceed- ings of the 34th ACM International Conference on I...
2025
-
[34]
Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, et al. 2025. Siren’s Song in the AI Ocean: A Survey on Hallucination in Large Language Models.Computational Linguistics51, 4 (2025), 1373–1418
2025
-
[35]
Yingli Zhou, Yaodong Su, Youran Sun, Shu Wang, Taotao Wang, Runyuan He, Yongwei Zhang, Sicong Liang, Xilin Liu, Yuchi Ma, et al. 2025. In-Depth Analysis of Graph-Based RAG in a Unified Framework.Proceedings of the VLDB Endowment 18, 13 (2025), 5623–5637
2025
Reviewed July 10, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.