REVIEW 3 major objections 5 minor 22 references
Reverse Engineering Compliance: A Dual-Graph Verification Framework for Auditing Legacy IT Security Concepts
T0 review · 3 major / 5 minor · reviewed 2026-07-10 · grok-4.5
Pith's one-line read Legacy IT security documents can be reverse-engineered into graphs and checked against real infrastructure before becoming machine-readable compliance artifacts.
desk verdict Solid reverse-path systems paper for IT-Grundschutz→OSCAL migration; the GGT-independence concern is real but already owned in Limitations, not a hidden collapse of the claim. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The dual-graph comparator that produces ΔG = (VO, VP, EO, ET, EG): node omissions, phantom nodes, edge omissions, topological conflicts, and ghost edges. Edges are checked only over matched nodes so a single missing asset is not counted again as many missing links.
What would settle it
On a held-out real IT security concept with a fully validated infrastructure inventory, inject known omissions, phantoms, and wrong links; if the five-class difference systematically misses them or reports large false differences when both sources are known consistent, the claim that ΔG makes document-infrastructure inconsistencies measurable fails.
Extended reading notes
Core claim
ASSERT makes inconsistencies between a legacy IT security concept and a verified infrastructure reference measurable by extracting an ontology-based document graph with full provenance, computing a five-class graph difference against an independent reference graph, and exporting schema-valid OSCAL system and assessment artifacts from the verified matched component.
Load-bearing premise
The independently built reference graph of the real infrastructure must be complete and correct; if it is wrong or incomplete, the measured differences will blame the document for errors that actually sit in the reference.
Editorial extensions
If this is right
- Organizations can migrate legacy security concepts into OSCAL without blindly importing outdated or invented assets.
- Audits gain a reproducible, class-labeled graph difference instead of relying only on free-text review or probabilistic model judgment.
- Schema-enforced extraction yields phantom-free system descriptions ready for export, at the cost of ignoring assets absent from the reference.
- Model choice and how much of the reference ontology is shown become explicit operational trade-offs between discovery and clean export.
- The exception list of unresolved mentions gives a concrete proxy for human review workload before final export.
Reading between the lines
- The same five-class difference could act as a continuous drift signal whenever infrastructure inventories and narrative security documents diverge over time.
- Cascaded modes (discovery first, then schema enforcement) may reduce the discovery-versus-clean-export trade-off without forcing one model to do both jobs.
- Provenance-linked graph differences could serve as audit evidence under other OSCAL-based control catalogs, not only the German Grundschutz path.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes ASSERT, a dual-graph framework for reverse-engineering legacy IT security concepts (IT-SCs) into machine-readable compliance artifacts. It extracts an attributed document graph GDoc from a legacy IT-SC via ontology-based LLM extraction with hierarchical provenance, deterministically compares GDoc to an independently reconstructed infrastructure reference graph GGT using a five-class cascading difference ΔG = (VO, VP, EO, ET, EG), and exports schema-valid NIST OSCAL v1.1.3 System Security Plan and Assessment Results from the verified matched component. Evaluation on the BSI RecPlast dataset compares a local open-weight model (Gemma 4 26B) and a commercial model (Claude Opus 4.7) under Generic, Schema-Guided, and Schema-Enforced ontology-exposure modes, with baseline extraction metrics (Table I) and class-specific F1 under single-fault and mixed fault injection (Table II). The central claim is that this architecture makes document–infrastructure inconsistencies measurable while exposing a trade-off between discovery of non-reference entities and phantom-free schema enforcement.
Significance. If the result holds, ASSERT fills a genuine reverse-path gap in compliance automation: most prior work generates or audits forward artifacts, whereas organizations must migrate large stocks of outdated narrative IT-SCs during the IT-Grundschutz to Grundschutz++ transition. Strengths include the clean separation of probabilistic extraction from deterministic set-theoretic verification, the cascading node-then-edge error taxonomy that avoids double-counting, explicit provenance to source paragraphs, and demonstrated schema-valid OSCAL SSP/AR export. The three-mode ablation and dual-model comparison make the discovery-vs-schema trade-off concrete and falsifiable. These are useful engineering contributions for Compliance-as-Code and regulated audit tooling, even if the empirical base remains a single public case study.
major comments (3)
- The load-bearing premise that GGT is an independent, verified source of truth is only partially secured. Background and Methodology reconstruct GGT from RecPlast intermediate structural-analysis artifacts and the SA agent of prior work [3]; Limitations correctly state that errors or incompleteness in GGT (including shadow IT) propagate directly into ΔG and that Schema-Enforced mode systematically ignores assets absent from GGT. On a single expert-generated public case, with no multi-source cross-validation of GGT, no seed-repeated runs, and no released reconstruction procedure, the evaluation cannot separate document error from reference incompleteness. Fault injection (Section V.D, Table II) only alters the IT-SC while holding GGT fixed, so it tests sensitivity under a perfect reference, not under the imperfect references the Limitations acknowledge. The central claim that ΔG attributes
- Table I and Table II together show that typed edge recall remains low (RE,r ≤ 0.261) across all configurations and that ET detection is highly model-dependent (Opus Schema-Guided F1 = 0.710 vs. Gemma 0.303; Gemma collapses under Vmixed). The paper attributes low edge recall to the narrative IT-SC stating only a subset of SA dependencies, which is plausible, but then the claim that ASSERT 'makes document-infrastructure inconsistencies measurable' is strongest for nodes and weakest for topology. The manuscript should either (a) quantify how much of the SA edge set is expressible in the IT-SC text under the chosen chunking, or (b) temper the topology-checking claim and treat edge metrics as secondary diagnostics rather than co-equal evidence of reverse topology-checking.
- Schema-Guided mode exhibits ontology-induced hallucination for Gemma (manual analysis: 12 of 16 unrecognized VO omissions reconstructed from GGT; VO F1 drops from 0.667 Generic to 0.258 Guided). This is an important negative result and is reported, but it is load-bearing for the claim that reference-ontology exposure is a controllable design knob. Without a quantitative leakage metric (e.g., fraction of extracted entities whose only support is the ontology prompt rather than the chunk) or an ablation that withholds entity names while retaining types, the three-mode comparison remains partly confounded by model instruction fidelity. A short leakage diagnostic or a stricter 'names-from-text-only' constraint would strengthen the mode-selection guidance in Fig. 2 and the Discussion.
minor comments (5)
- Notation for the matched edge sets (EM_GT,r, EM_Doc,r) and the five-tuple ΔG is introduced clearly in Background but is not restated with a single formal definition block; a short definition list would help readers navigating from Related Work to Methodology.
- Fig. 2 is evaluation-derived and useful, but the caption and body should state explicitly that the 'Best use / Strength / Problem' cells are qualitative summaries of Table I–II rather than independent measurements.
- The OSCAL export is described as a proof of concept with schema validation; reporting the exact schema version checks (NIST OSCAL v1.1.3) and whether any profile/catalog mismatches required local workarounds would improve reproducibility.
- Exception-list sizes (VExc up to 207) are given as a HITL workload proxy; a one-sentence note that actual review time was not measured (already in Limitations) could be moved earlier so readers do not over-interpret VExc as validated effort.
- Minor consistency: 'Grundschutz++' vs. 'IT-Grundschutz' and the arXiv date line (9 Jul 2026) should be checked against the intended publication timeline; also fix the occasional spacing in G GT / VM atched.
Circularity Check
Minor residual circularity only: Schema-Enforced forces VP=0 by construction (transparently reported) and GGT construction cites authors' prior SA agent; core dual-graph ΔG comparison and RecPlast evaluation remain independent.
-
self definitional
[V.C ASSERT Extraction Configurations (Schema-Enforced) and VI Results Table I]
"This dual strategy eliminates non-canonical phantom nodes (VP = 0) by construction. ... Schema-Enforced achieves high node-level F1,V = 0.957 for Gemma and F1,V = 0.985 for Opus and removes phantom nodes and ghost edges by construction, making it suitable for an OSCAL export."
In Schema-Enforced the prompt plus deterministic post-hoc filter against the GGT canonical list forces VP = 0 (and EG = 0) by definition of the mode. Consequently the reported PV = 1.000 and near-perfect F1,V are not independent extraction results; they are guaranteed by the filter. The paper is transparent about the construction and the discovery trade-off, so the circularity is mild and acknowledged rather than hidden.
-
self citation load bearing
[II.B Graph-Based Compliance Verification; IV.A Architecture; V.A Dataset]
"We describe the construction of G GT from those sources using a SA agent that has been validated in our prior work [3]. ... G GT is the verified infrastructure GT of the SA from [3]."
The central claim that ΔG attributes error to the legacy IT-SC (rather than to the reference) requires GGT to be an independent, verified source of truth. That independence is justified primarily by citation to the authors' own prior SA agent [3] and by reconstruction from RecPlast intermediate artifacts; the present paper supplies no multi-source cross-validation or released reconstruction code. Limitations correctly note that GGT errors propagate into ΔG. The self-citation is therefore load-bearing for the interpretation of ΔG, though the dual-graph comparator itself remains deterministic once GGT is given.
full rationale
ASSERT is an engineering framework paper, not a first-principles derivation of a physical or mathematical prediction. GDoc is built by LLM extraction from the legacy IT-SC (with provenance), then ΔG is computed by purely deterministic set operations (VO = VGT \ VDoc, VP = VDoc \ VGT, and cascading edge classes restricted to VMatched) against an external reference GGT. That comparison step does not reduce to its inputs by construction; the five error classes are definitional metrics, not fitted outputs renamed as discoveries. Fault-injection evaluation holds GGT fixed and alters only the IT-SC, measuring detection sensitivity under a controlled reference. The RecPlast public dataset further allows independent reconstruction of GGT from intermediate SA artifacts separate from the final IT-SC text. The only mild circular residues are (1) Schema-Enforced mode, which by design post-filters to the GGT canonical list so VP = EG = 0 and node precision is forced to 1.0 (the paper states this explicitly and discusses the discovery trade-off), and (2) the load-bearing premise that GGT is a verified external source of truth, which rests in part on the authors' prior SA agent [3] without multi-source cross-validation inside this paper. Limitations section already flags both issues. Neither residue collapses the central claim that the dual-graph comparator makes document-infrastructure inconsistencies measurable. Score 2 reflects one minor self-citation plus one transparent by-construction mode; the independent content of the reverse-path framework and deterministic ΔG remains intact. No fitted-parameter-as-prediction, uniqueness-theorem import, or ansatz-smuggling patterns appear.
Assumptions & free parameters
free parameters (3)
- fault_injection_k =
k=20; k′=5 per class
- chunk_size_and_split_policy =
≈3700 tokens average
- node_alignment_priority_rules =
3-level cascade
assumptions (4)
- domain assumption GGT reconstructed from SA intermediate artifacts is the independent trusted reference; document deviations are treated as document-side errors.
- domain assumption BSI IT-Grundschutz documents have naming consistency sufficient for strictly lexical node alignment without embedding similarity.
- ad hoc to paper Edge errors incident to omitted or phantom nodes should not be counted as independent topological errors (cascading evaluation over VM atched only).
- domain assumption Schema-valid OSCAL SSP/AR plus a minimal local profile/Assessment Plan is adequate evidence packaging when BSI provides catalog but no claimed-control profile.
invented entities (3)
-
ASSERT framework
-
Five-class compliance difference ΔG=(VO,VP,EO,ET,EG)
-
Generic / Schema-Guided / Schema-Enforced extraction modes
Cite this review
Pith. "Pith review of Reverse Engineering Compliance: A Dual-Graph Verification Framework for Auditing Legacy IT Security Concepts." pith.science (2026). https://pith.science/paper/UGH7EUPG
@misc{pith2026260708292,
author = {Pith},
title = {Pith review of: Reverse Engineering Compliance: A Dual-Graph Verification Framework for Auditing Legacy IT Security Concepts},
year = {2026},
howpublished = {\url{https://pith.science/paper/UGH7EUPG}},
note = {Machine review of arXiv:2607.08292}
}
read the original abstract
The NIS-2 Directive increases the need for continuous, auditable compliance evidence and motivates a shift from document-based compliance toward machine-readable compliance artifacts. The Open Security Controls Assessment Language (OSCAL) is a standard for this purpose, which the German Federal Office for Information Security (BSI) is adapting with Grundschutz++. However, companies are still managing extensive legacy IT security concepts (IT-SCs), and migrating them without verification could transfer outdated assets into the new format. While existing research primarily addresses the generation of new concepts, there is a lack of a verification framework that extracts legacy IT-SCs into an auditable intermediate representation, deterministically compares the extracted graph with an independently constructed reference state, and exports schema-valid OSCAL artifacts. This paper introduces the Automated Security Concept Structure Extraction and Reverse Topology-checking (ASSERT) Framework, which addresses this gap by using ontology-based extraction of legacy documents into formal document graphs, a five-class graph difference against a verified reference graph, and the export into schema-valid OSCAL outputs for system description and assessment evidence. Using the BSI's RecPlast dataset, we compare a local open-weight model and a commercial model across three configurations with different levels of reference-ontology exposure. The evaluation shows that ASSERT makes document-infrastructure inconsistencies measurable, but reveals a trade-off between discovering undocumented entities and enforcing a schema.
Figures
Reference graph
Works this paper leans on
-
[3]
L. R. Muth and M. Margraf, “Probabilistic agents in deterministic audits: Evaluating multi-agent systems for automated audits based on the German IT-Grundschutz,” inProc. 20th Annual IEEE International Systems Conference (SysCon), Halifax, NS, Canada, Apr. 2026, doi: https://doi.org/10.1109/SysCon66367.2026.11503560
-
[1]
Directive (EU) 2022/2555 (NIS-2 Directive),
European Parliament and Council, “Directive (EU) 2022/2555 (NIS-2 Directive),”Official Journal of the European Union, vol. L333, pp. 80– 152, Dec. 27, 2022. [Online]. Available: http://data.europa.eu/eli/dir/2 022/2555/oj
work page 2022
-
[2]
OSCAL: the Open Security Controls Assessment Language,
National Institute of Standards and Technology (NIST), “OSCAL: the Open Security Controls Assessment Language,” Mar. 26, 2026. [Online]. Available: https://pages.nist.gov/OSCAL
work page 2026
-
[4]
Y .-S. Hsia, F. Yu, and J.-H. R. Jiang, “Neuro-symbolic compliance: Integrating LLMs and SMT solvers for automated financial legal anal- ysis,” inProc. 2nd IEEE/ACM International Conference on AI-powered Software (AIware), Seoul, South Korea, pp. 1–10, Nov. 2025, doi: https://doi.org/10.1109/AIware69974.2025.00010
-
[5]
ComplianceNLP: Knowledge-Graph-Augmented RAG for Multi-Framework Regulatory Gap Detection
D. Guo, J. Wu, and S. M. Yiu, “ComplianceNLP: Knowledge-graph- augmented RAG for multi-framework regulatory gap detection,”Proc. ACL 2026 Industry Track, in press, arXiv:2604.23585, Apr. 2026, doi: https://doi.org/10.48550/arXiv.2604.23585
work page Pith review arXiv doi:10.48550/arxiv.2604.23585 2026
-
[6]
An empirical study of knowledge graph-enhanced RAG for information security compliance,
D. Jovanovski, M. Stojcheva, M. Dodevska, P. Lameski, I. Mishkovski, and D. Gjorgjevikj, “An empirical study of knowledge graph-enhanced RAG for information security compliance,”Information, vol. 17, no. 4, art. 389, Apr. 2026, doi: https://doi.org/10.3390/info17040389
-
[7]
PrivComp-KG: Leveraging KG and LLM for compliance verification,
L. Garza, L. Elluri, A. Piplai, A. Kotal, D. Gupta, and A. Joshi, “PrivComp-KG: Leveraging KG and LLM for compliance verification,” inProc. 2024 IEEE 6th International Conference on Trust, Privacy and Security in Intelligent Systems, and Applications (TPS-ISA), pp. 97–106, 2024, doi: https://doi.org/10.1109/TPS-ISA62245.2024.00021
-
[8]
P. Guldimann et al., “COMPL-AI framework: A technical interpretation and LLM benchmarking suite for the EU Artificial Intelligence Act,” arXiv:2410.07959, Oct. 2024, doi: https://doi.org/10.48550/arXiv.2410. 07959
Show all 22 references
- [9]
-
[10]
From COBIT to ISO 42001: Evaluating cyber- security frameworks for opportunities, risks, and regulatory compliance in commercializing large language models,
T. R. McIntosh et al., “From COBIT to ISO 42001: Evaluating cyber- security frameworks for opportunities, risks, and regulatory compliance in commercializing large language models,”Computers & Security, vol. 144, art. 103964, 2024, doi: https://doi.org/10.1016/j.cose.2024.103964
2024 doi
-
[11]
Can large language models reason and plan?,
S. Kambhampati, “Can large language models reason and plan?,”Annals of the New York Academy of Sciences, vol. 1534, no. 1, pp. 15–18, Mar. 2024, doi: https://doi.org/10.1111/nyas.15125
2024 doi
-
[12]
On the planning abilities of large language models: A critical investigation,
K. Valmeekam, M. Marquez, S. Sreedharan, and S. Kambhampati, “On the planning abilities of large language models: A critical investigation,” inProc. 37th International Conference on Neural Information Process- ing Systems (NeurIPS), New Orleans, LA, USA, art. 3320, pp. 75993– ...
2023 doi
-
[13]
CO2 (Co-compliance officer): An LLM-based ontology-driven methodology for generating knowledge graphs and AI compliance checking,
V . S. P. Turaga, T. Pahi, S. Tjoa, S. Siami-Namini, and A. Siami Namin, “CO2 (Co-compliance officer): An LLM-based ontology-driven methodology for generating knowledge graphs and AI compliance checking,”IEEE Access, 2025, doi: https://doi.org/10.1109/ACCE SS.2025.3639228
2025 doi
-
[14]
GraphCompliance: Aligning policy and context graphs for LLM-based regulatory compliance,
J. Chung et al., “GraphCompliance: Aligning policy and context graphs for LLM-based regulatory compliance,” arXiv:2510.26309, Oct. 2025, doi: https://doi.org/10.48550/arXiv.2510.26309
2025 doi
-
[15]
RAGulating Compliance: A multi-agent knowledge graph for regula- tory QA,
B. Agarwal, H. S. Jomraj, S. Kaplunov, J. Krolick, and V . Rojkova, “RAGulating Compliance: A multi-agent knowledge graph for regula- tory QA,” arXiv:2508.09893, Aug. 2025, doi: https://doi.org/10.48550 /arXiv.2508.09893
2025 arXiv
-
[16]
AgCyRAG: An agentic knowledge graph based RAG framework for automated security analysis,
K. Kurniawan, R. F. Ardian, E. Kiesling, and A. Ekelhart, “AgCyRAG: An agentic knowledge graph based RAG framework for automated security analysis,” inProc. Second International Workshop on Retrieval- Augmented Generation Enabled by Knowledge Graphs (RAGE-KG 2025), Nara, Japan...
2025
-
[17]
A case study for compliance as code with graphs and language models: Public release of the regulatory knowledge graph,
V . Ershov, “A case study for compliance as code with graphs and language models: Public release of the regulatory knowledge graph,” arXiv:2302.01842, Feb. 2023, doi: https://doi.org/10.48550/arXiv.2302. 01842
-
[18]
ForPKG: A framework for constructing forestry policy knowledge graph and application analysis,
J. Sun and Z. Luo, “ForPKG: A framework for constructing forestry policy knowledge graph and application analysis,” inProc. 2025 Inter- national Joint Conference on Neural Networks (IJCNN), pp. 1–8, 2025, doi: https://doi.org/10.1109/IJCNN64981.2025.11228178
2025 doi
-
[19]
CyberKG: Constructing a cybersecurity knowledge graph based on SecureBERT Plus for CTI reports,
B. Li, Q. Yang, C. Deng, and H. Pan, “CyberKG: Constructing a cybersecurity knowledge graph based on SecureBERT Plus for CTI reports,”Informatics, vol. 12, no. 3, art. 100, Sep. 2025, doi: https: //doi.org/10.3390/informatics12030100
2025 doi
-
[20]
Grounded knowledge graph extraction via LLMs: An anchor-constrained framework with provenance tracking,
Y . Yang, G. Chen, B. He, and Y . Zhao, “Grounded knowledge graph extraction via LLMs: An anchor-constrained framework with provenance tracking,”Computers, vol. 15, no. 3, art. 178, Mar. 2026, doi: https: //doi.org/10.3390/computers15030178
2026 doi
-
[21]
Hy- bridRAG: Integrating knowledge graphs and vector retrieval augmented generation for efficient information extraction,
B. Sarmah, B. Hall, R. Rao, S. Patel, S. Pasquali, and D. Mehta, “Hy- bridRAG: Integrating knowledge graphs and vector retrieval augmented generation for efficient information extraction,” arXiv:2408.04948, Aug. 2024, doi: https://doi.org/10.48550/arXiv.2408.04948
-
[22]
RecPlast GmbH
Bundesamt f ¨ur Sicherheit in der Informationstechnik (BSI), “RecPlast GmbH.” [Online]. Available: https://recplast-gmbh.de
Reviewed July 10, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.