Pith. sign in

REVIEW 30 references

Vector-based retrieval-augmented generation outperformed knowledge-graph RAG on SME cybersecurity questions across LLaMA, Mistral, and Qwen.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 01:48 UTC pith:K7ZIQ7Q3

arxiv 2608.00006 v1 pith:K7ZIQ7Q3 submitted 2026-06-08 cs.AI

Enhancing LLMs with Context-Specific Knowledge for Mitigating Misinformation in SMEs: A RAG-based Modeling and Analysis

Md. Samiul Islam , Iqbal H. Sarker , Chadni Islam , Ahmad Mohsin , Ahmed Ibrahim , Helge Janicke This is my paper
classification cs.AI
keywords llmsmisinformationhallucinationsmodelingsmesdecision-makingenvironmentsgeneration
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Large language models sometimes invent answers. Retrieval-Augmented Generation (RAG) tries to fix this by first searching a trusted document store and then asking the model to answer using only what was found. This paper compares two RAG styles on a small corpus of 31 Australian cyber-security and business documents: VectorRAG, which finds passages by meaning (embedding similarity), and GraphRAG, which builds a knowledge graph of entities and relationships and queries that graph.

The authors ran LLaMA, Mistral, and Qwen through both pipelines and report that VectorRAG scored higher on almost every metric — METEOR, completeness, F1-BERT, context relevance, and low 'hallucination risk' — while GraphRAG produced lower scores. They argue that graph retrieval needs dense relational structure, which this policy-oriented corpus lacks. They also show three examples where RAG corrects a base model's wrong answer, such as the correct ACSC cyber hotline number.

The qualitative conclusion is plausible and consistent with earlier RAG literature. However, the quantitative evidence is not reliable: the paper's central Table 4 lists identical values for Hallucination Risk and ROUGE-L Recall for every pipeline, two metrics that are defined differently. No code, dataset, or error bars are provided, and the best headline numbers are taken from different model configurations.

Core claim

The results demonstrate that RAG-enhanced LLMs can significantly improve response quality by reducing hallucinations and misinformation, thereby supporting more reliable, trustworthy, and context-aware decision-making in SME environments. More specifically: 'VectorRAG consistently outperformed GraphRAG in terms of answer relevance and completeness... vector-based pipelines maintained extremely low hallucination scores (0.0028–0.0073).' If the paper is correct, RAG grounding improves SME answers and vector retrieval is the better default on small, policy-oriented corpora.

Load-bearing premise

That the evaluation metrics in Table 4 are computed correctly and independently. Hallucination Risk and ROUGE-L Recall are defined as different quantities (unverifiable-content proportion vs. longest-common-subsequence recall), yet every pipeline in Table 4 shows identical values for the two columns (vector: 0.0028/0.0033/0.0073; graph: 0.0881/0.1053/0.1034). If the metric implementation simply copied one score, the paper's central quantitative comparison — including the headline 'hallucination rate of 0.28%' — is invalid. This is a measurement premise, not the central claim itself; the qualitative RAG-is-helpful direction could still be true even though the reported magnitudes cannot be trusted.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Axiom & Free-Parameter Ledger

8 free parameters · 3 axioms · 0 invented entities

The central claims rest on an unvalidated corpus, an implicit definition of hallucination risk, and automatic metrics treated as ground truth. These are domain assumptions rather than mathematical axioms. No new theoretical entities are introduced.

free parameters (8)
  • chunk_size_w = 700 tokens
    Hand-selected window size in Vector RAG chunking (Table 2, Algorithm 1); directly affects retrieval granularity and all downstream metrics.
  • chunk_overlap_o = 100 tokens
    Hand-selected overlap between chunks; affects context continuity and retrieval quality.
  • temperature = 0.7
    Generation temperature for vector RAG; influences response determinism and variability.
  • max_new_tokens = 512 (vector), 128 (graph)
    Output length cap; affects completeness and metric scores.
  • max_facts = 25
    Graph RAG retrieval cap; bounds facts passed to LLM.
  • max_sentences = 4
    Post-processing truncates Graph RAG answers to 4 sentences; affects all lexical/semantic metrics.
  • embedding_model = text-embedding-3-small
    Choice of embedding model determines vector retrieval; a free modeling choice.
  • top_k = not reported
    Algorithm 1 retrieves top-k chunks but k is never listed in Table 2; retrieval depth is a free parameter that strongly affects results.
axioms (3)
  • domain assumption The 31-PDF Australian corpus and its reference answers are a valid proxy for SME cybersecurity misinformation risk
    All conclusions about 'SME environments' generalize from one small, policy-oriented corpus.
  • domain assumption Hallucination Risk can be computed as the proportion of generated content unverifiable against retrieved context
    No formal definition or formula is given; this unstated metric definition is load-bearing for the headline hallucination reductions.
  • domain assumption Automatic metrics (METEOR, ROUGE-L, BERTScore) measure response quality and misinformation mitigation
    The paper uses these as proxies for 'useful response generation' and 'contextual relevance' without validation against human judgments; only a few anecdotal examples are presented.

pith-pipeline@v1.3.0-alltime-deepseek · 15654 in / 11839 out tokens · 112191 ms · 2026-08-04T01:48:08.941715+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Enhancing LLMs with Context-Specific Knowledge for Mitigating Misinformation in SMEs: A RAG-based Modeling and Analysis." pith.science (2026). https://pith.science/paper/K7ZIQ7Q3

@misc{pith2026260800006,
  author       = {Pith},
  title        = {Pith review of: Enhancing LLMs with Context-Specific Knowledge for Mitigating Misinformation in SMEs: A RAG-based Modeling and Analysis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/K7ZIQ7Q3}},
  note         = {Machine review of arXiv:2608.00006}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Large Language Models (LLMs), a part of artificial intelligence (AI), are increasingly being adopted by Small and Medium Enterprises (SMEs) to enhance question-answering capabilities and support business decision-making processes. However, hallucinations in LLM-generated outputs can serve as a source of misinformation, reducing user confidence in their reliability and trustworthiness within SMEs. Retrieval-Augmented Generation (RAG) has emerged as a promising approach to address this challenge by incorporating external knowledge sources into the modeling process. In this paper, we present VectorRAG and GraphRAG modeling approaches to mitigate hallucinations and misinformation risks and evaluate their effectiveness in SME environments. Our experimental evaluation is conducted on multiple state-of-the-art LLMs, including LLaMA, Mistral, and Qwen, to assess performance in terms of useful response generation, risk of hallucination, contextual relevance, as well as human-interpretation. The results demonstrate that RAG-enhanced LLMs can significantly improve response quality by reducing hallucinations and misinformation, thereby supporting more reliable, trustworthy, and context-aware decision-making in SME environments.

Figures

Figures reproduced from arXiv: 2608.00006 by Ahmad Mohsin, Ahmed Ibrahim, Chadni Islam, Helge Janicke, Iqbal H. Sarker, Md. Samiul Islam.

Figure 1
Figure 1. Figure 1: An illustration of the RAG-based system architecture The dataset also incorporated international policy re￾ports, SME industry surveys, cybersecurity resilience stud￾ies, governance reports, and peer-reviewed academic liter￾ature to strengthen the broader contextual and operational coverage of the corpus. These documents contributed in￾sights into SME digital transformation, financial resilience, cybersecu… view at source ↗
Figure 2
Figure 2. Figure 2: Effectiveness of LLMs: Vector RAG vs. Graph RAG [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Response Generation using Vector RAG most relevant chunks, as shown in [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Illustration of constructed KG for SMEs. linked within the knowledge graph [PITH_FULL_IMAGE:figures/full_fig_p011_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: An illustration of representing knowledge considering several entities in KG [PITH_FULL_IMAGE:figures/full_fig_p012_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

30 extracted references · 1 canonical work pages · 1 internal anchor

  1. [1]

    A. G. Report, Communications, media, a. s. b. & ombudsman, f. e. a report on small business matters, communications and media, australian small business and family enterprise ombudsman, 2023

    2023

  2. [2]

    W. Reim, P. Yli-Viitala, J. Arrasvuori, V. Parida, Tackling business model challenges in sme internationalization through digitalization, 2022

    2022

  3. [3]

    T. Tan, A. Thirunavukarasu, J. Campbell, P. Keane, L. Pasquale, M.Abramoff,J.Kalpathy-Cramer,F.Lum,J.Kim,S.Baxter,D.Ting, Generative artificial intelligence through chatgpt and other large lan- guagemodelsinophthalmology:Clinicalapplicationsandchallenges, Ophthalmology Science 3 (2023)

    2023

  4. [4]

    S.Menten,A.Smits,R.A.Kok,K.Lauche,M.vanGils, Externalre- sourcing for digital innovation in manufacturing smes, Technovation 140 (2025) 103142

    2025

  5. [5]

    I. H. Sarker, Llm potentiality and awareness: a position paper from theperspectiveoftrustworthyandresponsibleaimodeling, Discover Artificial Intelligence 4 (2024) 40

    2024

  6. [6]

    Alansari, H

    A. Alansari, H. Luqman, Large language models hallucination: A comprehensivesurvey, ComputerScienceReview61(2026)100970

    2026

  7. [7]

    Z. Ji, N. Lee, R. Frieske, T. Yu, D. Su, Y. Xu, E. Ishii, Y. J. Bang, A. Madotto, P. Fung, Survey of hallucination in natural language generation, ACM Comput. Surv. 55 (2023) 1–38

    2023

  8. [8]

    Lavrinovics, R

    E. Lavrinovics, R. Biswas, J. Bjerva, K. Hose, Knowledge graphs, large language models, and hallucinations: An nlp perspective, Jour- nal of Web Semantics 85 (2025) 100844

    2025

  9. [9]

    Kasneci, K

    E. Kasneci, K. Sessler, S. Küchemann, M. Bannert, D. Dementieva, F. Fischer, U. Gasser, G. Groh, S. Günnemann, E. Hüllermeier, S.Krusche,G.Kutyniok,T.Michaeli,C.Nerdel,J.Pfeffer,O.Poquet, M. Sailer, A. Schmidt, T. Seidel, M. Stadler, J. Weller, J. Kuhn, G. Kasneci, Chatgpt for good? on opportunities and challenges of large language models for education, L...

    2023

  10. [10]

    M. A. Uddin, I. H. Sarker, Fine-tuning vs rag: A position paper from theperspectiveofllm-basedcybersecuritymodeling, Computing&AI Connect 3 (2026) 1–10

    2026

  11. [11]

    I. H. Sarker, H. Janicke, A. Mohsin, L. Maglaras, Sme-team: lever- aging trust and ethics for secure and responsible use of ai and llms in smes, npj Artificial Intelligence 2 (2026) 12

    2026

  12. [12]

    W.Zhang,J.Zhang, Hallucinationmitigationforretrieval-augmented large language models: A review, Mathematics 13 (2025) 856

    2025

  13. [13]

    Computing 28 (2025) 27

    R.Upadhyay,M.Viviani,Enhancinghealthinformationretrievalwith rag by prioritizing topical relevance and factual accuracy, Discov. Computing 28 (2025) 27

    2025

  14. [14]

    Farquhar, J

    S. Farquhar, J. Kossen, L. Kuhn, et al., Detecting hallucinations in large language models using semantic entropy, Nature 630 (2024) 625–630

    2024

  15. [15]

    Islam and Sarker et al.:Preprint submitted to ElsevierPage 14 of 15 VectorRAG and GraphRAG based Modeling

    V.S.Pendyala,C.E.Hall, Explainingmisinformationdetectionusing large language models, Electronics 13 (2024) 1673. Islam and Sarker et al.:Preprint submitted to ElsevierPage 14 of 15 VectorRAG and GraphRAG based Modeling

    2024

  16. [16]

    A. Adel, N. Alani, Can generative ai reliably synthesise literature? exploring hallucination issues in chatgpt, AI & SOCIETY 40 (2025) 6799–6812

    2025

  17. [17]

    B. Hu, Q. Sheng, J. Cao, Y. Li, D. Wang, Llm-generated fake news induces truth decay in news ecosystem: A case study on neural news recommendation, 2025. doi:10.48550/arXiv.2504.20013. arXiv:2504.20013

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2504.20013 2025

  18. [18]

    Försch, J

    T.Han,S.Nebelung,F.Khader,T.Wang,G.Müller-Franzes,C.Kuhl, S. Försch, J. Kleesiek, C. Haarburger, K. K. Bressem, J. N. Kather, D. Truhn, Medical large language models are susceptible to targeted misinformation attacks, NPJ Digital Medicine 7 (2024) 288

    2024

  19. [19]

    M.Y.Mohammed,S.A.Ali,S.K.Ali,etal.,Aftina:enhancingstabil- ity and preventing hallucination in ai-based islamic fatwa generation using llms and rag, Neural Computing and Applications 37 (2025) 20957–20982

    2025

  20. [20]

    Z. Li, Z. Wang, W. Wang, K. Hung, H. Xie, F. L. Wang, Retrieval- augmented generation for educational application: A systematic sur- vey, Computers and Education: Artificial Intelligence 8 (2025) 100417

    2025

  21. [21]

    Bratanic, Graph algorithms for data science: With examples in Neo4j, Simon and Schuster, 2024

    T. Bratanic, Graph algorithms for data science: With examples in Neo4j, Simon and Schuster, 2024

    2024

  22. [22]

    A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. de las Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, L.R.Lavaud,M.-A.Lachaux,P.Stock,T.L.Scao,T.Lavril,T.Wang, T. Lacroix, W. E. Sayed, Mistral 7b, 2023.arXiv:2310.06825

    Pith/arXiv arXiv 2023

  23. [23]

    W. Yang, L. Some, M. Bain, B. Kang, A comprehensive survey on integrating large language models with knowledge-based methods, Knowledge-Based Systems 318 (2025) 113503

    2025

  24. [24]

    Team, Qwen3 technical report, 2025

    Q. Team, Qwen3 technical report, 2025

    2025

  25. [25]

    Mudzakir, J

    A. Mudzakir, J. T. Ativiirya Janaprasetya, E. Kayowuan, V. J. Lewi Engel,B.Saputro,Evaluatingthreelargelanguagemodelsforsecurity incident detection and analysis, Procedia Computer Science 269 (2025) 465–473. The 10th International Conference on Computer Science and Computational Intelligence 2025

    2025

  26. [26]

    A. G. et al., The llama 3 herd of models, 2024

    2024

  27. [27]

    Al Faraby, A

    S. Al Faraby, A. Romadhony, Adiwijaya, Analysis of llms for educational question classification and generation, Computers and Education: Artificial Intelligence 7 (2024) 100298

    2024

  28. [28]

    J.S.Erickson,H.Santos,V.Pinheiro,J.P.McCusker,D.L.McGuin- ness, Llm experimentation through knowledge graphs: Towards improvedmanagement,repeatability,andverification,JournalofWeb Semantics 85 (2025) 100853

    2025

  29. [29]

    Sequeda, D

    J. Sequeda, D. Allemang, B. Jacob, Knowledge graphs as a source of trust for llm-powered enterprise question answering, Journal of Web Semantics 85 (2025) 100858

    2025

  30. [30]

    P. Bedi, A. Thukral, S. Dhiman, Xlr-kgdd: Leveraging llm and rag for knowledge graph-based explainable disease diagnosis using mul- timodalclinicalinformation, KnowledgeandInformationSystems67 (2025) 7451–7471. Islam and Sarker et al.:Preprint submitted to ElsevierPage 15 of 15

    2025