REVIEW 3 major objections 6 minor 25 references
An MCP-grounded multi-agent pipeline converts natural-language critical-infrastructure descriptions into source-verified knowledge graphs and audit-ready OSCAL plans without active scanning, concentrating residual error in a single reviewab
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · grok-4.5
2026-07-10 09:58 UTC pith:5XY6S7JV
load-bearing objection Careful OT compliance PoC that relocates LLM failure to Phase-0 extraction; the single synthetic plant is the real ceiling, and the authors already say so. the 3 major comments →
From Legacy Documentation to OSCAL: An MCP-Based Agent Pipeline for Threat-Informed Continuous Compliance in Critical Infrastructure
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
Deterministic MCP retrieval against authoritative cyber-threat sources prevents fabricated vulnerabilities and scores in the knowledge graph, while residual failure is dominated by upstream semantic mistakes when entities are extracted from unstructured text. In the WaterWork water-utility scenario the pipeline achieves 0.90 CVE recall and 1.00 D3FEND recall, zero factual hallucination on deterministically sourced nodes, and schema-valid OSCAL SSP and SAR outputs; a single mis-extracted generic Windows entity accounts for the observed contextual false positives.
What carries the argument
The eight-phase MCP-grounded pipeline that separates probabilistic LLM stages (entity extraction, rehabilitation, relevance review, assessment writing) from deterministic retrieval stages (CVE discovery, CRH scoring, triage, taxonomy expansion), together with the Critical Infrastructure Relevance Heuristic that multiplies a CVSS floor by asset criticality and exposure then adds EPSS and KEV bonuses for safety-first triage.
Load-bearing premise
All reported recall, precision and error-shift results rest on a single synthetic water-utility architecture with a hand-curated ground-truth CVE list, so the reliability story may not transfer to real, messier legacy documentation or other sectors.
What would settle it
Execute the same pipeline on two or more independently documented real or independently constructed critical-infrastructure sites, recompute CVE and D3FEND recall against expert ground truth, and test whether Semantic Hallucination Rate and Contextual False Positive Rate remain concentrated in Phase-0 entity extraction at comparable magnitudes; if factual hallucinations reappear or taxonomy and triage errors dominate, the central reliability claim fails.
If this is right
- Operators can produce audit-ready OSCAL SSP and SAR documents from existing prose documentation without scanning fragile OT networks.
- Residual risk becomes localized to a short, human-reviewable entity list before expensive deterministic enrichment runs.
- Low per-run cost enables continuous re-assessment that mitigates temporal gaps in threat-intelligence feeds.
- The same ingestion layer can feed machine-readable control catalogs once they stabilize, closing the gap between legacy docs and automated compliance.
Where Pith is reading between the lines
- In practice the decisive control point will almost always be a human-in-the-loop review immediately after Phase 0 rather than further downstream generation improvements.
- The same error-shift pattern—stochastic inventory mistakes that propagate as real but irrelevant findings—should appear in any domain that extracts structured asset lists before deterministic enrichment (medical devices, software BOMs, facility inventories).
- Raising Phase-0 entity precision above roughly 95 percent would be expected to reduce contextual false-positive rate roughly in proportion, offering a clear quantitative target for the next iteration.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes an eight-phase multi-agent pipeline that converts unstructured natural-language OT/critical-infrastructure documentation (plus passive Shodan input) into a source-verified threat knowledge graph and NIST OSCAL System Security Plan and Security Assessment Report artifacts, without active scanning. LLM phases are restricted to entity extraction, triage rehabilitation, relevance review, and SAR generation; CVE discovery, CRH scoring, triage thresholds, and taxonomy expansion are deterministic and mediated by 15 MCP servers over authoritative CTI sources. On a single synthetic water-utility scenario (WaterWork) with a manually curated ground truth of 292 CVEs, 15 ATT&CK techniques, and 34 D3FEND mappings, five runs report CVE recall 0.90, D3FEND recall 1.00, 0% factual hallucination on deterministically sourced KG nodes, OSCAL schema validity, and residual error concentrated in Phase-0 entity extraction (Semantic Hallucination Rate 12.5%, Contextual False Positive Rate ~8.5%). A transparent OT-oriented Critical infrastructure Relevance Heuristic (CRH) combining CVSS, EPSS, KEV, asset criticality, and exposure is introduced for triage.
Significance. If the result holds beyond the synthetic setup, the work is a useful systems contribution for non-invasive, threat-informed continuous compliance in OT: it shows how MCP tool-use can enforce source-verified CTI retrieval, produce schema-valid OSCAL SSP/SAR, and convert distributed LLM fabrication risk into a localized, human-reviewable Phase-0 control point. Strengths that should be credited include the explicit factual vs. semantic/contextual error decomposition, five-run stability metrics (Jaccard ~0.95), external-source verification of KG nodes, strict OSCAL v1.1.2 schema validation, one-at-a-time CRH sensitivity with 100% ground-truth retention under multi-rule triage, and an honest limitations section. The architecture fills a real gap between legacy NL documentation and machine-readable compliance artifacts that prior OSCAL and MCP-security work does not jointly address.
major comments (3)
- §IV.B–C and §V: All quantitative support for the central reliability claim (CVE/D3FEND recall, 0% factual hallucination, error shift to Phase 0, CRH triage behavior, OSCAL audit-readiness) rests on a single author-curated synthetic architecture (WaterWork) and one manually verified ground truth. The manuscript acknowledges this in §VI, but the abstract, title, and contribution bullets still present the pipeline as enabling continuous automated compliance for critical infrastructure in general. Either add at least one additional evidence-based scenario (different sector/topology/documentation style) or substantially qualify transfer claims so that reported recalls and the Phase-0 control-point narrative are scoped as single-scenario PoC evidence.
- §IV.A.3 Eqs. (1)–(4) and §V: CRH is described as an “empirically grounded, domain-specific heuristic,” yet the free parameters (SCADA floor 5.0, C_crit/E_exp multipliers, EPSS×5, KEV +5, U thresholds, triage rules) are stated as “empirically derived” without a derivation dataset, fitting procedure, or practitioner study. The SSVC comparison (ρ=0.797) is correctly labeled indirect because both share CVSS/EPSS/KEV; it cannot substitute for external validation of the OT-specific multipliers. Either provide the empirical basis for the constants, report a structured expert concordance study, or reframe CRH as a transparent, adjustable safety-first policy whose parameters are disclosed rather than empirically validated.
- §V and §VI: The paper’s core reliability insight is that residual risk concentrates in Phase-0 entity extraction and is “suitable for a time-efficient manual review.” No human-in-the-loop experiment, review-time measurement, or inter-annotator protocol is reported; the 12.5% SHR / ~8.5% Contextual FP Rate is measured only against author ground truth. Because Phase 0 is explicitly the single LLM-dependent trust boundary and the claimed operational mitigation, the manuscript should either evaluate a review step (even a small expert review of extracted entities/CPEs) or weaken the claim that residual risk is operationally manageable without additional evidence.
minor comments (6)
- Abstract and §I: “perfect D3FEND recall” / “1.00” is accurate for the ground-truth set of 34 mappings, but §V notes taxonomy-chain completeness of only 0.16 vs 2.80 because only the top-25 high-urgency CVEs are fully expanded. State this design cap when advertising perfect D3FEND recall so readers do not infer full-graph coverage.
- §IV.A.1 / Fig. 1: Phase numbering mixes 0, 0b, 3a, 3b, 5, 6 (agent) with 1–4 (deterministic); Phase 0b is mentioned in the architecture paragraph but not defined in the phase list. Align figure labels and text.
- §V: Report absolute TP/FP/FN counts alongside recall/precision for CVE, ATT&CK, and D3FEND so that the effect of the generic “Windows (unknown version)” entity and the 29 missed CVEs is fully reconstructible without reverse-engineering percentages.
- §II.C: The claim that the deterministic pipeline configuration acts as an “implicit, reproducible SAP” is reasonable but should briefly note how an auditor would recover the SAP parameters (thresholds, MCP sources, taxonomy rules) from the released artifacts.
- Presentation: occasional doubled words and phrasing (“operational operational technology,” “critical infrastructure. critical infrastructure sectors”); normalize terminology (OT vs operational technology) and fix minor grammar throughout.
- §III.A: Related MCP security work is well covered; a short explicit comparison table (input type, OT support, OSCAL output, hallucination control) would make the novelty claim against AgCyRAG and attack-graph–OSCAL work easier to audit.
Circularity Check
No significant circularity: main claims are measured against external CTI sources and an independently curated ground truth, not forced by definition or self-citation.
full rationale
This is an empirical systems paper, not a first-principles derivation. The load-bearing claims (0% factual hallucination on KG nodes, CVE recall 0.90, D3FEND recall 1.00, schema-valid OSCAL SSP/SAR) are checked against external authoritative sources (NVD, CPE, ICS-CERT, D3FEND, NIST OSCAL JSON schemas) and a manually curated WaterWork ground truth of 292 CVEs / 15 attack paths / 34 D3FEND mappings. MCP grounding constrains retrieval by design, but success is not defined as matching the pipeline’s own outputs; nodes are verified to exist in primary sources, and residual error (SHR 12.5%, contextual FP ~8.5%) is reported against that same external inventory. CRH is an explicit, transparent heuristic with disclosed parameters, not a fit renamed as prediction; the CRH–SSVC Spearman comparison is secondary and the authors themselves label it an indirect consistency check because both share CVSS/EPSS/KEV inputs. There is no self-definitional loop, no uniqueness theorem imported from the authors, no load-bearing self-citation chain, and no renaming of a known result as a new derivation. Single-scenario evaluation limits transfer, but that is a scope/correctness concern, not circularity.
Axiom & Free-Parameter Ledger
free parameters (7)
- CRH SCADA floor on CVSS (S_base min) =
5.0
- Asset criticality multipliers C_crit =
1.5 / 1.25 / 1.0
- Exposure multipliers E_exp =
1.0 / 1.5
- EPSS scale factor and KEV bonus in B_threat =
EPSS×5; KEV +5
- Urgency category and triage thresholds =
U>12; role U>5; bands 22/14/7
- Phase-1 strategy confidence scores =
1.0 / 0.9 / 0.7 / 0.5
- Taxonomy expansion caps =
top 25 CVEs; ≤20 chains
axioms (6)
- domain assumption Active vulnerability scanning of legacy OT/ICS is often operationally unacceptable; assessment must proceed from passive artifacts and documentation.
- domain assumption Deterministic MCP tool calls against authoritative CTI APIs yield verifiable ground truth for CVE/CWE/CAPEC/ATT&CK/D3FEND nodes, eliminating factual identifier/score hallucination.
- domain assumption Purdue-model proximity to physical process is an appropriate basis for asset criticality weights in OT vulnerability urgency.
- domain assumption Safety-first OT triage should prioritize recall over precision; false negatives are more costly than false positives.
- ad hoc to paper A single incorrectly extracted entity is acceptable if residual risk remains visible and human-reviewable because operators typically know versions/OS/assets.
- standard math Standard graph/set evaluation metrics (recall, precision, Jaccard, Spearman ρ) and NIST OSCAL JSON Schema validity are adequate success criteria for audit-readiness claims.
invented entities (3)
-
Critical infrastructure Relevance Heuristic (CRH)
no independent evidence
-
Eight-phase MCP-grounded multi-agent pipeline + seven-node OT threat KG schema
no independent evidence
-
WaterWork synthetic evidence-based reference architecture and curated ground truth
no independent evidence
read the original abstract
In critical infrastructure, operational technology environments often cannot be actively scanned, and yet active system feedback is needed for risk assessment and compliance. This paper presents a non-invasive, MCP-grounded multi-agent pipeline that converts natural-language system descriptions into source-verified knowledge graph and audit-ready artifacts in the NIST OSCAL format for continuous automated compliance management. The architecture decouples LLM-based reasoning from deterministic knowledge retrieval against authoritative threat-intelligence sources, reducing the risk of fabricated vulnerabilities and hallucinated attack paths. In an evidence-based synthetic scenario of a water utility, the pipeline achieves 0.90 CVE recall and perfect D3FEND recall. It generates a schema-valid OSCAL System Security Plan and an OSCAL Security Assessment Report. Nevertheless, the core insight is not that grounding via MCP eliminates errors (e.g., hallucinations) entirely from the pipeline, but that it shifts errors into the first phase of asset extraction from the natural language description. Here, a single incorrectly extracted entity can lead to genuine but irrelevant CVEs in subsequent stages of the pipeline, which consumes time and resources. However, it makes the remaining risk visible, verifiable, and suitable for a time-efficient manual review, since the infrastructure (e.g., version numbers, OS, etc.) is typically known.
Figures
Reference graph
Works this paper leans on
-
[1]
Disrupting the First Reported AI- Orchestrated Cyber Espionage Campaign
Anthropic Threat Intelligence, “Disrupting the First Reported AI- Orchestrated Cyber Espionage Campaign”, Anthropic News, Nov. 13, 2025. [Online]. Available: https://www.anthropic.com/news/ disrupting-AI-espionage/. (Accessed: 2026-03-08)
work page 2025
-
[2]
OpenAI, “GPT-4.1 Model”, OpenAI API Documentation. [Online]. Available: https://platform.openai.com/docs/models/gpt-4.1. (Accessed: 2026-03-08)
work page 2026
-
[3]
A Survey of Cyber- Physical Attacks and Detection Methods in Smart Water Distribution Systems
H. H. Addeen, Y . Xiao, J. Li, and M. Guizani, “A Survey of Cyber- Physical Attacks and Detection Methods in Smart Water Distribution Systems”, IEEE Access, vol. 9, July 2021. Available: https://doi.org/10. 1109/ACCESS.2021.3095713
-
[4]
European Parliament and Council of the European Union, “Directive (EU) 2022/2555 of the European Parliament and of the Council of 14 December 2022 on measures for a high common level of cybersecurity across the Union, amending Regulation (EU) No 910/2014 and Directive (EU) 2018/1972, and repealing Directive (EU) 2016/1148 (NIS 2 Directive)”, Official Jour...
work page 2022
-
[5]
Bundesamt f ¨ur Sicherheit in der Informationstechnik (BSI), “Stand-der- Technik-Bibliothek”, GitHub, Sep. 2025. [Online]. Available: https:// github.com/BSI-Bund/Stand-der-Technik-Bibliothek. (Accessed: 2026- 03-08)
work page 2025
-
[6]
The Linux Foundation, “Model Context Protocol”, GitHub repository, 2024, Available: https://github.com/modelcontextprotocol. (Accessed: 2026-03-08)
work page 2024
-
[7]
Towards the Automation of Attack Graph- Based Risk Assessment with OSCAL
I. Koufos, M. Christopoulou, G. Xilouris, M.-A. Kourtis, M. Sou- valioti, and P. Trakadas, “Towards the Automation of Attack Graph- Based Risk Assessment with OSCAL”,Lecture Notes in Networks and Systems, pp. 319–328, Mar. 2025. Available: https://doi.org/10.1007/ 978-3-031-76459-2 30
work page 2025
-
[8]
Model Context Protocol (MCP): Landscape, Security Threats, and Future Research Directions
X. Hou, Y . Zhao, S. Wang, and H. Wang, “Model Context Protocol (MCP): Landscape, Security Threats, and Future Research Directions”, CoRR, abs/2503.23278, Mar. 2025. Available: https://doi.org/10.48550/ arXiv.2503.23278
work page internal anchor Pith review Pith/arXiv arXiv 2025
-
[9]
Integrated Risk Scoring and Exploit Prediction for Cyber-Physical Power System Vul- nerabilities
F. Kausar, L. Batiste, A. Muallem, and S. Hussain, “Integrated Risk Scoring and Exploit Prediction for Cyber-Physical Power System Vul- nerabilities”, Energy Informatics, Feb. 2026. Available: https://doi.org/ 10.1186/s42162-026-00640-x
-
[10]
CISA Stakeholder-Specific Vulnerability Categorization Guide
“CISA Stakeholder-Specific Vulnerability Categorization Guide”, Cybersecurity and Infrastructure Security Agency, Nov. 2022. Available: https://www.cisa.gov/sites/default/files/publications/cisa-ssvc-guide% 20508c.pdf
work page 2022
-
[11]
Common Vulnerability Scoring System Version 4.0: User Guide
FIRST, “Common Vulnerability Scoring System Version 4.0: User Guide”, Document Version 1.2, Nov. 1, 2023. [Online]. Available: https://www.first.org/cvss/v4.0/user-guide. (Accessed: 2026-03-08)
work page 2023
-
[12]
V . M. Vilches et al., “Towards an open standard for assessing the severity of robot security vulnerabilities, the Robot Vulnerability Scoring System (RVSS)”, Pre-print, arXiv, Nov. 2021. Available: https://doi.org/ 10.48550/arXiv.1807.10357
work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.1807.10357 2021
-
[13]
N. Shimizu, and M. Hashimoto, “Vulnerability Management Chaining: An Integrated Framework for Efficient Cybersecurity Risk Prioritiza- tion”, IEEE Access, vol. 14, pp. 31407–31424, Feb. 2026. Available: https://doi.org/10.1109/ACCESS.2026.3665768
-
[14]
Dynamic Vulnerability Severity Cal- culator for Industrial Control Systems
P. Cheimonidis, and K. Rantos, “Dynamic Vulnerability Severity Cal- culator for Industrial Control Systems”, International Journal of In- formation Security, vol. 23, pp. 2655–2676, 2024. Available: https: //doi.org/10.1007/s10207-024-00858-4
-
[15]
A Survey on Vulnerability Prioritization: Taxonomy, Metrics, and Research Challenges
Y . Jiang, N. Oo, Q. Meng, H. W. Lim, and B. Sikdar, “A Survey on Vul- nerability Prioritization: Taxonomy, Metrics, and Research Challenges”, Pre-print, arXiv, Feb. 2025. Available: https://arxiv.org/abs/2502.11070
work page internal anchor Pith review Pith/arXiv arXiv 2025
-
[17]
Exploit Prediction Scoring System (EPSS)
J. Jacobs, S. Romanosky, B. Edwards, I. Adjerid, and M. Roytman, “Exploit Prediction Scoring System (EPSS)”,Digital Threats: Research and Practice, vol. 2, no. 3, pp. 1–17, 2021. Available: https://dl.acm. org/doi/10.1145/3436242
-
[18]
MCP Safety Audit: LLMs with the Model Context Protocol Allow Major Security Exploits
B. Radosevich and J. Halloran, “MCP Safety Audit: LLMs with the Model Context Protocol Allow Major Security Exploits”, CoRR, abs/2504.03767, Apr. 2025. Available: https://doi.org/10.48550/arXiv. 2504.03767
work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv 2025
-
[19]
AgCyRAG: an Agentic Knowledge Graph based RAG Framework for Automated Secu- rity Analysis
K. Kurniawan, R. Firdaus, E. Kiesling, and A. Ekelhart, “AgCyRAG: an Agentic Knowledge Graph based RAG Framework for Automated Secu- rity Analysis”, Workshop (RAGE-KG 2025 at ISWC 2025), vol. 4079, Nov. 2025. Available: https://ceur-ws.org/V ol-4079/paper11.pdf
work page 2025
-
[20]
T. A. Syed, M. R. Belgaum, S. Jan, A. A. Khan, and S. S. Alqahtani, “Agentic AI for Autonomous Defense in Software Supply Chain Secu- rity: Beyond Provenance to Vulnerability Mitigation”, Pre-print, arXiv, Dec. 2025. Available: https://arxiv.org/abs/2512.23480
-
[21]
Enterprise-Grade Security for the Model Context Protocol (MCP): Frameworks and Mitigation Strategies
V . S. Narajala, and I. Habler, “Enterprise-Grade Security for the Model Context Protocol (MCP): Frameworks and Mitigation Strategies”, Pre- print, arXiv, Apr. 2025. Available: https://arxiv.org/abs/2504.08623
work page internal anchor Pith review Pith/arXiv arXiv 2025
-
[22]
M. Bhatt, V . S. Narajala, and I. Habler, “ETDI: Mitigating Tool Squatting and Rug Pull Attacks in Model Context Protocol (MCP) by using OAuth- Enhanced Tool Definitions and Policy-Based Access Control”, Pre-print, arXiv, Jun. 2025. Available: https://arxiv.org/abs/2506.01333
work page internal anchor Pith review Pith/arXiv arXiv 2025
-
[23]
P. Ackerman, “Industrial Cybersecurity”, 2nd ed. Birmingham, UK: Packt Publishing, 2024
work page 2024
-
[24]
Using LLMs for Security Advisory Investigations: How Far Are We?
B. F. Abdullah, Y . S. Nugroho, B. Reid, R. G. Kula, K. Shimari, and K. Matsumoto, “Using LLMs for Security Advisory Investigations: How Far Are We?”, Pre-print, arXiv, June 2025. Available: https://arxiv.org/ abs/2506.13161v1
work page internal anchor Pith review Pith/arXiv arXiv 2025
-
[25]
Cybersecurity and Infrastructure Security Agency (CISA), “Binding Operational Directive 22-01: Reducing the Significant Risk of Known Exploited Vulnerabilities”, Nov. 2021. Available: https://www.cisa.gov/ binding-operational-directive-22-01. (Accessed: 2026-03-08)
work page 2021
-
[26]
Guide to Industrial Control Systems (ICS) Security
K. Stouffer, V . Pillitteri, S. Lightman, M. Abrams, and A. Hahn, “Guide to Industrial Control Systems (ICS) Security”, National Institute of Standards and Technology, NIST Special Publication 800-82 Rev. 2, 2015
work page 2015
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.