REVIEW 5 major objections 6 minor 25 references
The paper claims that a category-guided hybrid of dense retrieval and knowledge-graph reasoning—CatRAG—raises a regulation-answering chatbot to 98.89% F1, beating plain RAG.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-04 12:48 UTC pith:KEBOP3QD
load-bearing objection CatRAG is a sensible hybrid design and the new Vietnamese regulation dataset is a real asset, but the headline 98.89% F1 is likely measuring in/out-of-domain scope, not answer quality. the 5 major comments →
REBot: From RAG to CatRAG with Semantic Enrichment and Graph Routing
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The central claim is that adding a category-guided graph branch to standard RAG improves answer quality for domain-specific questions. The knowledge graph is built in two passes: chunks are embedded and stored in a vector index, NER extracts entities that are linked to chunks, an LLM extracts typed relations, and a classifier attaches each chunk to one of five regulation subgraphs. At query time the same classifier labels the question; the system then retrieves top-k chunks from the global vector store and from the matching subgraph, expands the graph chunks to entities and relations, and passes the combined context to an LLM. The paper reports that this construction yields 98.89% F1 with a
What carries the argument
The load-bearing component is the CatRAG-Query algorithm, supported by the Regulation-Enriched Graph (REG), a labelled graph G=(E,R) whose entities come from NER and whose typed relations come from LLM-based extraction, partitioned into category subgraphs by the classifier. The query algorithm embeds and classifies the question, retrieves top-k chunks from both the vector store and the predicted subgraph, expands those chunks to entities and relations, and feeds the unified context to the LLM. Graph retrieval is the differentiator: it supplies surrounding structured evidence that dense retrieval alone does not provide, while the category partition keeps that evidence focused on the right dom
Load-bearing premise
The load-bearing assumption is that the F1 score of 98.89% actually measures answer quality; the evaluation dataset only marks each query as belonging to the regulation domain or not, and the paper gives no protocol for scoring generated answers, so if F1 simply reflects in-domain versus out-of-domain discrimination, the 'accurate answers' claim is unsupported.
What would settle it
Regenerate answers for a sample of the 1,313 evaluation pairs, have human judges or gold answers grade correctness independent of query scope, and recompute precision/recall/F1; if the score drops to at or below the plain-RAG variant, the category-guided graph claim is falsified. A more refined check: take in-domain queries and deliberately corrupt the graph subgraph or classifier label; if answer quality does not degrade, the graph branch is not carrying the reported benefit.
If this is right
- For one of the two tested language models, CatRAG outperforms plain RAG at every threshold from 0.6 to 0.8, with the largest margin at 0.8 (F1 91.05 vs 89.19) and the highest absolute F1 of 98.89 at 0.6.
- Adding graph-based retrieval costs roughly two seconds per query (about 7.3 s vs 5.1 s for one model), which the paper still calls acceptable for real-time advising.
- A lightweight classifier suffices for routing: the paper's best classifier reaches about 95.7% F1, and that accuracy is enough to yield the reported CatRAG gains.
- The framework relies on compact language models rather than large ones, so the same category-guided recipe could be reproduced in other low-resource domains.
- The paper's own qualitative review says answers are 'largely accurate and contextually enriched,' positioning CatRAG as a practical alternative to pure RAG for institutions with fragmented policy documents.
Where Pith is reading between the lines
- Editorial inference: the reported advantage of CatRAG over RAG is model-dependent on the paper's own numbers—for the second model at threshold 0.6, plain RAG actually scores higher (98.95 F1 vs 98.73), so the headline 'CatRAG consistently outperforms RAG' would not survive across every model/threshold combination.
- Editorial inference: because the evaluation dataset labels queries only as in-domain or out-of-domain, the 98.89% F1 may be measuring retrieval routing ability rather than answer correctness; a direct test would regrade outputs against gold answers or a human rubric before claiming factual superiority.
- Editorial inference: the category-routing idea transfers naturally to other fragmented, policy-heavy domains (health, legal, financial), where a small taxonomy is known and documents are unstructured; a testable extension is replacing the fixed five-category schema with a learned hierarchy.
- Editorial inference: the paper's own listed limitations—excessive context causing incoherent responses, hard long-context classification, and ~7-second latency—suggest the next bottleneck is context control, so adding reranking or selective entity expansion is the natural next experiment.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes REBot, an academic-regulation chatbot for Can Tho University, powered by CatRAG, a hybrid retrieval-augmented generation and knowledge-graph framework. CatRAG combines dense vector retrieval with a category-guided knowledge graph: documents are chunked, embedded, classified into five regulation domains, and linked to entities and relations in a Neo4j graph; at query time a fastText classifier routes the query to the corresponding subgraph, whose top chunks and entities are merged with vector retrieval results and passed to an LLM. The authors construct two Vietnamese datasets (a Q&A evaluation set and a classification training set), report classification results for fastText versus traditional baselines, compare RAG and CatRAG across thresholds and two LLMs, and report a headline F1-score of 98.89% for CatRAG with gpt-4o-mini at threshold 0.6. They also present a web application and qualitative examples.
Significance. If the evaluation were sound, CatRAG would be a useful and practical contribution: it provides a reusable recipe for combining category-guided graph retrieval with dense retrieval in a low-resource language (Vietnamese), includes a concrete knowledge-graph construction algorithm and query algorithm, offers complexity analysis for both, and releases data and code publicly. The paper also honestly discusses latency and context-bloat limitations. The architecture is not circular; the concern is whether the reported quantitative results measure what they claim. The headline number, however, rests on an evaluation protocol that is not described, and the similarity threshold is tuned on the same evaluation set. These issues are load-bearing for the central claim of state-of-the-art performance with an F1-score of 98.89%.
major comments (5)
- [Section 5.1, Table 1 and Section 5.2, Table 2] The F1 metric is not interpretable as an answer-quality measure. The Q&A dataset is labeled only as 'Truth - Knowledge related to CTU academic regulation' (909 samples) versus 'Other - Knowledge outside of CTU academic regulation' (404 samples). No gold answers, human rubric, LLM-judge procedure, exact-match criterion, or any other grading protocol is given for computing Precision, Recall, and F1 in Table 2. As written, the reported 98.89% F1 could simply reflect in-domain/out-of-domain discrimination of the response (i.e., whether the model's answer stays within the regulation scope), not whether the answer is factually correct or complete. This invalidates the headline claim as stated. The authors must specify the exact evaluation protocol, provide the grading rubric or judge prompts, and ideally report per-question accuracy on the 909 in-domain items separately.
- [Section 5.2, grid search over threshold] The similarity threshold is fitted on the same evaluation set used to report the headline result. The text says 'we conduct a gridsearch from 0.1 to 1.0. Then, [0.6,0.8] obtained the high accuracy,' and the abstract and Table 2 report the best threshold (0.6) as the system's performance. This is a selection artifact: with 10 thresholds explored, the maximum F1 is expected to be optimistic relative to out-of-sample performance. The paper should use a validation split for threshold selection or report the full threshold sweep with a clear train/validation/test separation. The statement in Section 5.2 that 'a similarity threshold of 0.7 is applied' also conflicts with the headline 0.6 result; please clarify which threshold is actually used in the deployed system.
- [Table 1 and Section 5.1] The dataset sizes are internally inconsistent: the text reports 1,319 Q&A pairs and 3,256 classification questions, while Table 1 totals 1,313 and 3,252 respectively. The arithmetic in Table 1 is correct for its rows (474+611+655+477+485+550 = 3,252; 909+404 = 1,313), so the discrepancy is in the prose or in the table labels. This matters because the classification and evaluation sets are the only empirical basis for the paper's claims; the authors should correct the numbers and report the exact split used for training and testing the classifier.
- [Section 5.2, Table 2 discussion] The narrative claim that CatRAG 'consistently outperforms its RAG counterpart across all threshold settings' is contradicted by the Mistral row at threshold 0.6, where RAG achieves F1 98.95 vs CatRAG 98.73, and the Mistral row at threshold 0.7 has nearly equal recall. Please rephrase the claim to describe the actual pattern, and report whether the differences are statistically significant under any reasonable test (e.g., paired bootstrap or McNemar). Without significance testing, the 0.22-point F1 gain for gpt-4o-mini at 0.6 may be noise.
- [Section 5.2 / Conclusion] The paper calls the result 'state-of-the-art performance' but provides no external baselines from the literature, no comparison with prior academic-advising chatbots, and no baseline beyond the paper's own RAG variant. Given the evaluation metric is undefined (see first comment), the SOTA claim is unsupported. At minimum, the authors should compare against a simple fastText-routing plus vector-retrieval baseline, and against a standard GraphRAG baseline, with the same evaluation protocol.
minor comments (6)
- [Section 4, Algorithms 1 and 2] The algorithm listings contain LaTeX artifacts: 'v_q leftarrow f_emb(q)' and similar arrows are typeset with a leading minus sign, making the code difficult to read. Please fix the arrow commands (e.g., leftarrow without the hyphen).
- [Section 3 / Section 5.3] The framework description says NER and relation extraction use Underthesea, but the paper does not specify which NER model or relation-extraction prompt is used, nor the accuracy of the NER step. A short description or reference would help reproducibility.
- [Table 2 / Table 3] The caption for Table 2 lists 'Eval. Time' with only two cells filled; other rows show '–'. Please either report the time for all rows or remove the column and describe the timing measurement separately. Table 3 reports 'average of 10 runs' but does not state the standard deviation or the number of queries used; please add this.
- [Section 5.3] The text says 'we distilled four key questions' but only two examples are shown. Either show all four or rephrase to 'two illustrative examples'.
- [Abstract and Section 2.5] The abstract states that CatRAG 'integrates RAG with GraphRAG' but elsewhere GraphRAG is described as a technique that 'often underperforms standard RAG' [7]. The framing is mildly inconsistent; please clarify whether the paper's contribution is a new variant of GraphRAG or a union of RAG and GraphRAG.
- [Throughout] There are several typographical issues: 'T ogether' (Section 4), inconsistent comma use around 'e.g.,', and the repeated footnote markers 2/5 and 3/8. A careful proofreading pass would improve readability.
Circularity Check
No construction-level circularity; the reported F1 is an evaluation result with validity caveats, not a derivation from its own inputs.
full rationale
Walking the claimed derivation chain: CatRAG-Construct (Algorithm 1) and CatRAG-Query (Algorithm 2) are constructive algorithms whose outputs—a vector store/graph and a generated response—depend on inputs such as chunks, an embedding function, a classifier, a knowledge graph, and an LLM. No output quantity is defined in terms of the headline F1, and the headline F1 is not fed back into the algorithms. The F1=98.89 in Table 2 is an evaluation result of one configuration (threshold 0.6) and is not a fitted prediction derived from the model equations. The threshold grid search reported in Section 5.2 ('we conduct a gridsearch from 0.1 to 1.0. Then, [0.6,0.8] obtained the high accuracy') is a model-selection/validity concern, not a circularity: it does not make the architecture's derivation equivalent to its inputs. Similarly, the absence of a grading protocol for the Table 1 labels ('Truth 909 — Knowledge related to CTU academic regulation; Other 404 — Knowledge outside of CTU academic regulation') is a benchmark-validity issue, not a circular reduction. The only self-citation, [15], appears in the introduction as background context ('an LLM-powered chatbot that offers CTU students [15] fast, accurate, and up-to-date answers') and is not load-bearing: no uniqueness theorem or ansatz is imported from it. The paper also reports honest limitations in Section 5.4, including excessive graph context and latency, and compares CatRAG against its own plain-RAG baseline. Because the central architecture and its reported comparison are independently grounded in the algorithms and measurements, no specific circular step can be exhibited, and the circularity score is 0.
Axiom & Free-Parameter Ledger
free parameters (5)
- KG similarity threshold τ =
0.6 (best of grid search over [0.1, 1.0])
- k_vec (top vector chunks) =
5
- k_graph (top graph chunks) =
not specified
- fastText hyperparameters =
epochs=100, ngrams=3
- chunk size =
not specified
axioms (5)
- domain assumption The expert-authored Q&A dataset (labels 'Truth 909' / 'Other 404') and its gold answers are correct and complete.
- domain assumption The 250-node knowledge graph extracted from CTU PDFs via Docling, Underthesea NER, and LLM relation extraction faithfully represents the regulations.
- domain assumption PhoBERTv2 embeddings with cosine similarity at threshold 0.6 yield semantically valid matches for Vietnamese regulation text.
- domain assumption The fastText classifier (95.67% F1) routes queries to the correct subgraph often enough that routing errors do not materially degrade the QA result.
- domain assumption gpt-4o-mini / mistral-small-2506 generate faithful answers from the retrieved context.
read the original abstract
Academic regulation advising is essential for helping students interpret and comply with institutional policies, yet building effective systems requires domain specific regulatory resources. To address this challenge, we propose REBot, an LLM enhanced advisory chatbot powered by CatRAG, a hybrid retrieval reasoning framework that integrates retrieval augmented generation with graph based reasoning. CatRAG unifies dense retrieval and graph reasoning, supported by a hierarchical, category labeled knowledge graph enriched with semantic features for domain alignment. A lightweight intent classifier routes queries to the appropriate retrieval modules, ensuring both factual accuracy and contextual depth. We construct a regulation specific dataset and evaluate REBot on classification and question answering tasks, achieving state of the art performance with an F1 score of 98.89%. Finally, we implement a web application that demonstrates the practical value of REBot in real world academic advising scenarios.
Figures
Reference graph
Works this paper leans on
-
[1]
High-Confidence Computing4(2), 100211 (2024)
A survey on large language model (llm) security and privacy: The good, the bad, and the ugly. High-Confidence Computing4(2), 100211 (2024)
2024
-
[2]
Opdahl, A., Veres, C.: Named entity ex- traction for knowledge graphs: A literature overview
Al-Moslmi, T., Gallofré Oca˜ na, M., L. Opdahl, A., Veres, C.: Named entity ex- traction for knowledge graphs: A literature overview. IEEE Access8, 32862–32881 (2020)
2020
-
[3]
Journal of Big Data9(1), 10 (2022)
Chiche, A., Yitagesu, B.: Part of speech tagging: a systematic review of deep learn- ing and machine learning approaches. Journal of Big Data9(1), 10 (2022)
2022
-
[4]
Hue University Journal of Science: Techniques and Technology 131, 39–52 (06 2023) 12 Thanh Ma et al
Doan, H., Le, V., van, K.: Xây dựng khung ứng dụng ai chatbot trong lĩnh vực quy chế đào tạo. Hue University Journal of Science: Techniques and Technology 131, 39–52 (06 2023) 12 Thanh Ma et al
2023
-
[5]
arXiv preprint arXiv:2310.04560 (2023)
Fatemi, B., Halcrow, J., Perozzi, B.: Talk like a graph: Encoding graphs for large language models. arXiv preprint arXiv:2310.04560 (2023)
Pith/arXiv arXiv 2023
-
[6]
Computer Science Review29, 21–43 (2018)
Goyal, A., Gupta, V., Kumar, M.: Recent named entity recognition and classifica- tion techniques: a systematic review. Computer Science Review29, 21–43 (2018)
2018
-
[7]
graphrag: A systematic evaluation and key insights
Han, H., Shomer, H., Wang, Y., Lei, Y., Guo, K., Hua, Z., Long, B., Liu, H., Tang, J.: Rag vs. graphrag: A systematic evaluation and key insights. arXiv preprint arXiv:2502.11371 (2025)
arXiv 2025
-
[8]
arXiv preprint arXiv:2501.00309 (2024)
Han, H., Wang, Y., Shomer, H., Guo, K., Ding, J., Lei, Y., Halappanavar, M., Rossi, R.A., Mukherjee, S., Tang, X., et al.: Retrieval-augmented generation with graphs (graphrag). arXiv preprint arXiv:2501.00309 (2024)
Pith/arXiv arXiv 2024
-
[9]
ACM Com- puting Surveys54(4), 1–37 (Jul 2021)
Hogan, A., Blomqvist, E., Cochez, M., et al, D.: Knowledge graphs. ACM Com- puting Surveys54(4), 1–37 (Jul 2021)
2021
-
[10]
arXiv preprint arXiv:2405.16506 (2024)
Hu, Y., Lei, Z., Zhang, Z., Pan, B., Ling, C., Zhao, L.: Grag: Graph retrieval- augmented generation. arXiv preprint arXiv:2405.16506 (2024)
Pith/arXiv arXiv 2024
-
[11]
Proceedings of Vietnamese Speech and Language Processing (VLSP) (2016)
Huyen, N.T.M., Luong, V.X.: Vlsp 2016 shared task: Named entity recognition. Proceedings of Vietnamese Speech and Language Processing (VLSP) (2016)
2016
-
[12]
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., K¨ uttler, H., Lewis, M., tau Yih, W., Rockt¨ aschel, T., Riedel, S., Kiela, D.: Retrieval-augmented generation for knowledge-intensive nlp tasks (2021)
2021
-
[13]
ACM Transactions on Intelligent Systems and Technology (TIST)13(2), 1–41 (2022)
Li, Q., Peng, H., Li, J., Xia, C., Yang, R., Sun, L., Yu, P.S., He, L.: A survey on text classification: From traditional to deep learning. ACM Transactions on Intelligent Systems and Technology (TIST)13(2), 1–41 (2022)
2022
-
[14]
Luong, H., Luong, K.: A chatbot-based academic advising model for student in informationtechnology:Acasestudy.SaudiJournalofEngineeringandTechnology 10(3), 93–100 (2025)
2025
-
[15]
In: International Conference on Intelligent Systems and Data Science
Ma, T., Chau, T.K., Thai, P.A., Tram, T.M., Huynh, K., Tran-Nguyen, M.T.: Racos: Ai-routed chat-voice admission consulting support system. In: International Conference on Intelligent Systems and Data Science. pp. 295–310. Springer (2024)
2024
-
[16]
Minh, P.Q.N.: A feature-rich vietnamese named-entity recognition model (2018), https://arxiv.org/abs/1803.04375
Pith/arXiv arXiv 2018
-
[17]
Expert Systems with Applications106, 36–54 (2018)
Miro´ nczuk, M.M., Protasiewicz, J.: A recent overview of the state-of-the-art ele- ments of text classification. Expert Systems with Applications106, 36–54 (2018)
2018
-
[18]
Meta-Radiology1(2), 100022 (2023)
Nazir, A., Wang, Z.: A comprehensive survey of chatgpt: Advancements, applica- tions, prospects, and challenges. Meta-Radiology1(2), 100022 (2023)
2023
-
[19]
arXiv preprint arXiv:2003.00744 (2020)
Nguyen, D.Q., Nguyen, A.T.: Phobert: Pre-trained language models for viet- namese. arXiv preprint arXiv:2003.00744 (2020)
Pith/arXiv arXiv 2003
-
[20]
arXiv preprint arXiv:1711.04951 (2017)
Nguyen, D.Q., Vu, T., Nguyen, D.Q., Dras, M., Johnson, M.: From word segmen- tation to pos tagging for vietnamese. arXiv preprint arXiv:1711.04951 (2017)
Pith/arXiv arXiv 2017
-
[21]
IEEE Transactions on Knowledge and Data Engineering36(7), 3580–3599 (2024)
Pan, S., Luo, L., Wang, Y., Chen, C., Wang, J., Wu, X.: Unifying large language models and knowledge graphs: A roadmap. IEEE Transactions on Knowledge and Data Engineering36(7), 3580–3599 (2024)
2024
-
[22]
Pande, A., Patil, R., Mukkemwar, R., Panchal, R., Bhoite, S.: Comprehensive study of google gemini and text generating models: Understanding capabilities and performance (11 2024)
2024
-
[23]
Artificial Intelligence Review56(11), 13071–13102 (2023)
Peng, C., Xia, F., Naseriparsa, M., Osborne, F.: Knowledge graphs: Opportunities and challenges. Artificial Intelligence Review56(11), 13071–13102 (2023)
2023
-
[24]
arXiv preprint arXiv:2307.07697 (2023)
Sun, J., Xu, C., Tang, L., Wang, S., Lin, C., Gong, Y., Ni, L.M., Shum, H.Y., Guo, J.: Think-on-graph: Deep and responsible reasoning of large language model on knowledge graph. arXiv preprint arXiv:2307.07697 (2023)
Pith/arXiv arXiv 2023
-
[25]
ICAIIS’20 pp
Yao, T., Zhai, Z., Gao, B.: Text classification model based on fasttext. ICAIIS’20 pp. 154–157 (2020)
2020
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.