Machine Learning-Based Security Policy Analysis

Amir Eaman; Joann Sum; Krish Jain; Pranav Kapoor

REVIEW 4 major objections 5 minor 15 references

Machine Learning-Based Security Policy Analysis

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Graph-based machine learning automates SELinux policy analysis, with an MLP neural network reaching 95% accuracy.

desk verdict The paper is an honest but flawed proof-of-concept: the headline 95% accuracy is a selection artifact from co-adapting the class taxonomy to model errors on the same dataset. read the letter →

arxiv 2501.00085 v2 pith:374HTUOF submitted 2024-12-30 cs.LG cs.AIcs.CR

Krish Jain , Joann Sum , Pranav Kapoor , Amir Eaman This is my paper · ORCID

classification cs.LGcs.AIcs.CR

keywords SELinuxpolicyanalysismandatoryaccesscontrolgraph-basedmachinelearningNode2vecanomalydetectionneuralnetworksecurity

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Security-Enhanced Linux (SELinux) enforces mandatory access controls through a policy language so intricate that manual analysis is error-prone and existing tools produce hard-to-interpret abstractions. This paper tries to establish that policy analysis can be automated by converting SELinux type-enforcement rules into a graph, embedding the graph with Node2vec, and feeding those embeddings into standard classifiers. The authors report that a Multi-Layer Perceptron (MLP) neural network detects policy violations with 95% accuracy, balanced precision/recall of 0.95/0.97/0.95, outperforming Random Forest (93%) and SVM (92%). If the result holds, administrators could audit SELinux policies automatically and catch misconfigurations such as separation-of-duty breaks and domain-transition issues without combing through thousands of rules by hand.

What carries the argument

The central machinery is the graph model of SELinux type enforcement: types become nodes and allow rules become edges, stored in the Neo4j graph database. Node2vec generates continuous vector embeddings that preserve local and global graph structure through flexible random walks, and these embeddings are the input to Random Forest, SVM, and MLP classifiers. The paper also relies on a refined taxonomy of ten violation classes, consolidated from an initial sixteen, which it argues balances specificity with generalization for detection.

What would settle it

Train the pipeline on the paper's synthetic set, then run it on a real production SELinux policy whose violations have been labeled by independent SELinux experts; if accuracy on that held-out policy falls to near chance levels, the reported generalization claim fails. A cheaper check is to retrain with the 16-class scheme while holding out entire violation classes during training and measuring recall on those held-out classes.

Watch

Extended reading notes

Core claim

The paper claims that representing SELinux type-enforcement rules as a graph, with subject, object, and class nodes connected by allow-rule edges, and then learning node embeddings via Node2vec, yields feature vectors that preserve policy neighborhoods and violation patterns. On a synthetic dataset of 455 policy rules spanning ten violation classes, the MLP classifier reaches 95% accuracy with a macro F1 of 0.95, the best of the three models tested. The authors further claim that this graph-based approach significantly outperforms existing analysis techniques by capturing relational structure that prior machine-learning attempts missed.

Load-bearing premise

The 95% accuracy is an unbiased estimate of real-world detection performance, yet the evaluation uses a self-constructed synthetic dataset whose ten violation classes were iteratively redefined based on how well the models scored on that same data, with no held-out real-policy test.

Editorial extensions

If this is right

Automated policy auditing becomes feasible: the graph-embedding pipeline can flag candidate violations in policies with thousands of rules, reducing the manual inspection burden on administrators.
The MLP's consistent edge over Random Forest and SVM across dataset sizes (125, 401, and 455 rules) indicates that the neural net benefits most from the graph-derived embeddings.
The trajectory from 16 classes down to 10 shows that overly fine-grained violation categories hurt detection of broader patterns, making the consolidated taxonomy part of the claimed performance.
The same graph-embedding pipeline is claimed to generalize across policy aspects, including transition graphs, attribute graphs, and object-class graphs, suggesting one framework can cover multiple SELinux rule types.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The authors' iterative refinement of violation classes based on model performance on the same dataset risks circularity: part of the reported 95% accuracy may reflect classes chosen to be easy for the models rather than intrinsically detectable violations.
A stronger test would freeze the class taxonomy before model development and evaluate on a separate, independently labeled real policy; the paper reports no such held-out test.
The graph representation could transfer to other mandatory-access-control systems, such as AppArmor or Windows integrity levels, where policies also form subject-object-permission relations, though the violation taxonomy would need re-deriving.
Because the dataset is synthetic and server-focused, the next validation step implied by the paper's own future-work section is measuring performance on production policies with hundreds of thousands of rules.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

The headline number is not trustworthy, but the paper is not a throwaway. The authors build a graph representation of SELinux policy rules, embed it with Node2vec, and run RF/SVM/MLP classifiers on a self-constructed synthetic dataset of 455 rules. That specific combination is new in the SELinux tooling space and it is a reasonable proof-of-concept. They also deserve credit for documenting the iterative taxonomy refinement in Section IV.C: starting with 5 classes, splitting and merging based on model errors, ending at 10. That transparency is what lets a reader see the central problem.

The soft spot is load-bearing. The 95% accuracy reported in Table 5 is a selection artifact. The final 10-class schema was chosen after observing which classes the models confused—consolidating "overly permissive" into "improper privilege," merging contradictory-rule subclasses, and grouping four access-missing classes because recall was 0.25. The same 455-rule dataset was then used to report the final accuracy. There is no described train/test split, no cross-validation, and no independent held-out set. So the numbers measure how well the models fit a taxonomy that was tuned to their mistakes on that exact data. Real-policy generalization is untested. The abstract's claim of "significantly outperforming existing analysis techniques" is also unsupported: no SETools, SPRT, or other baselines were run.

The reliance on an unpublished, self-cited manuscript [9] for the graph model is a secondary concern. It may be fine, but as submitted it is not independently checkable.

None of this means the direction is wrong. The violation taxonomy, while repackaging known SELinux issues, is sensible, and the idea of learning embeddings over policy graphs is plausible. The paper is an honest proof-of-concept that needs a rigorous evaluation: held-out policies, real production policies, and comparison against existing tools.

My recommendation: send it to peer review rather than desk reject. A serious referee would demand that evaluation, and the authors seem capable of providing it. As submitted, the central claim does not hold.

Referee Report

4 major / 5 minor

Summary. The paper proposes a graph-based machine learning pipeline for detecting violations in SELinux policies. Policies are represented as Neo4j graphs, embedded with Node2vec, and classified by Random Forest, SVM, and MLP models. On a synthetic dataset of 455 policy rules, the MLP reaches 95% accuracy with macro precision/recall/F1 of 0.95/0.97/0.95, and the authors conclude that the approach automates SELinux policy analysis and significantly outperforms existing techniques. The study also reports experiments with 5, 10, and 16 violation classes, documenting how the class schema was iteratively refined.

Significance. If the reported results were reliable, the paper would be a useful step toward automated SELinux policy auditing, and the comparison of three classifiers on graph embeddings is a reasonable research question. The authors deserve credit for building a concrete pipeline (Neo4j + Node2vec + ML classifiers) and for being unusually transparent about the iterative refinement of their violation taxonomy. However, the central accuracy claim is undermined by a fundamental methodological flaw: the violation classes were adjusted based on model performance on the same dataset used to report the final accuracy, and no held-out or real-policy evaluation is provided. The dependence on an unpublished self-cited manuscript for the graph model, and the absence of code, data, and baseline comparisons, further reduce the contribution to an unvalidated proof-of-concept.

major comments (4)

[Section IV.C, Tables 2–5] The reported accuracies (including the abstract's 95% MLP figure) are not unbiased estimates of detection performance, because the violation-class taxonomy was iteratively adjusted based on model performance on the same 455-rule dataset. The text documents that the initial 5 classes were split due to high false positives, that classes were expanded to 16 when SoD/Contradictory/Missing rules underperformed, and that classes were then consolidated to 10 because the 'models struggled with overly specific access violations' and showed low recall for missing-rule classes (Table 4). Thus the final Table 5 numbers measure how well the models fit a taxonomy optimized to their errors on the same data, not the detection of real SELinux misconfigurations.
[Section IV.B] The evaluation is performed solely on a self-constructed synthetic dataset of 455 policy rules derived from Fedora 39 and Ubuntu Server templates. The paper does not describe any train/test split, cross-validation procedure, or held-out set, so the reported accuracies could reflect memorization rather than generalization. In addition, no comparison is made with SETools, SPRT, SMT-based verification, or formal-analysis tools, leaving the abstract's claim that the method 'significantly outperform[s] existing analysis techniques' without empirical support.
[Section IV.C, Table 4] The per-class recall for 'Missing necessary file access for system processes' is 0.25 for Random Forest and MLP and 0.06 for SVM. The paper's narrative states that this observation drove the consolidation of four separate access-violation classes into a single class. This is direct evidence that the final 10-class schema was chosen in response to the models' weaknesses on the very data used to report the final accuracy, which constitutes selection on the test set and invalidates the headline numbers.
[Section III.B and reference [9]] The graph model is based on an unpublished, self-cited manuscript by Eaman et al. [9], so the reader cannot verify the representation or assess its novelty relative to prior work. The paper also does not report the Node2vec hyperparameters, the MLP architecture, or the training procedure, and no source code or synthetic dataset is provided. These omissions are load-bearing because the entire pipeline depends on those embedding and training choices, making the reported results irreproducible.

minor comments (5)

[Section IV.A] Listing 5 is introduced as 'ten distinct violation classes' but shows eleven numbered entries (0 through 10), and the mapping from these numbers to the classes discussed in the text is confusing; clarify the numbering and reconcile it with the subsequent tables.
[Section IV.C] There is a typographical error: 'a balanced amount of each violatioin class' should read 'violation class'.
[Tables 1–5] The tables render with broken lines inside numeric values (e.g., '0.7 9' and '0.8 5'), which appears to be a formatting defect; fix the table layout so that each metric is readable.
[Abstract and Section I] The phrase 'significantly outperforming existing analysis techniques' is used in the abstract and introduction, but no comparative experiments against existing tools are presented anywhere in the manuscript; either add such a comparison or weaken the claim.
[References] Reference [4] is a YouTube video, which is an unconventional citation for a technical claim about SELinux; consider replacing it with a primary source. Also, the reference list order and formatting are inconsistent with a standard IEEE style.

Circularity Check

2 steps flagged · score 6.0 of 10

Central accuracy claim rests on a performance-fitted violation taxonomy evaluated on the same synthetic dataset, alongside a load-bearing unpublished self-citation.

fitted input called prediction [Section IV.C, 'Results' (Tables 4-5 and Listing 5)]
"Our refinement from 16 to 10 classes involved strategic consolidation of related violation types. The overly permissive access class (originally class 2) was merged into improper privilege assignment, as our models showed significant overlap in detecting these patterns. Similarly, we consolidated all contradictory rule violations (original classes 5, 12, and 13) into a single 'Contradictory type transitions' class..."

The final 10-class taxonomy that produces the headline 95% MLP accuracy in Table 5 was selected after the authors observed model errors, overlaps, and low recall on the exact 455-rule dataset used for that table. The paper explicitly says the refinement was driven by the models struggling with the previous class definitions. Consequently, the reported accuracy measures how well the models fit a label schema that was itself engineered around model mistakes on the evaluation data, rather than measuring detection of an independently fixed set of SELinux violations. The abstract's claim of 95% accuracy for 'detecting policy violations' therefore partially reduces to a selection artifact: the class boundaries are fitted to the test data by construction.
self citation load bearing [Section III.B, Graph Model for SELinux Policies]
"Our graph representation of SELinux policies in Figure 1 builds on the model proposed by Eaman et al. [9], which defines three fundamental node types to capture policy relationships."

The graph representation that drives the entire Node2vec embedding and downstream ML pipeline is justified solely by reference [9], 'Graph-Powered Mining and Refining of SELinux Security Policies,' unpublished, 2024, whose first author is also an author of this paper. No external verification, formal proof, or independent benchmark is supplied for this representation. This is a load-bearing self-citation because the central modeling choice rests on an unpublished manuscript from the same group; however, this particular step does not by itself force the accuracy numbers, so it contributes to the paper's fragility but not to the direct label-fitting reduction.

full rationale

The strongest circularity is not in the ML training itself but in the evaluation and labeling protocol. Section IV.C documents an iterative process: the authors started with 5 classes, split classes because of false positives, expanded to 16, and then consolidated to the final 10 based on observed model confusion and low recall on the same 455-rule dataset. The final Table 5 accuracies (RF 0.93, SVM 0.92, MLP 0.95) are reported on that same dataset using the performance-engineered taxonomy. Thus the 'prediction' of violation classes is partly a measure of how well the models fit class boundaries chosen after seeing model outputs on the test data. This is a form of fitted input called prediction, and it undermines the abstract's claim of 'significantly outperforming existing analysis techniques,' since no held-out real-policy evaluation or comparison to SETools, SPRT, or formal methods is reported. Additionally, the graph model is grounded in an unpublished self-citation by one of the paper's authors, which is load-bearing for the method but not for the numerical results. Because the central claim does reduce in part to the label-engineering loop, the circularity score is 6 rather than lower; it is not 8-10 because the embeddings and model training still involve genuine computation and the paper is transparent about the iterative refinement.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the untested representativeness of the synthetic dataset, the choice of violation classes tuned to model performance, and unverified embedding quality. These are domain assumptions rather than invented entities.

free parameters (4)

Violation class schema = 10 refined classes
The number and definitions of violation classes were iteratively adjusted (5, 10, 16, then 10) based on model accuracy, and the final accuracy depends on this choice (Section IV.C).
Node2vec hyperparameters = not reported
Embedding dimension, walk length, number of walks, and context window are not specified; these affect the quality of the input features (Section IV).
MLP architecture and hyperparameters = not reported
Number of layers, hidden units, learning rate, and regularization are unspecified, so the reported 95% cannot be reproduced (Section IV.C).
Synthetic dataset composition = 455 policy rules
The dataset was hand-constructed to represent each violation class, with no release or independent validation (Section IV.B).

assumptions (4)

domain assumption Type Enforcement rules can be faithfully represented as Subject-Object-Class graphs
Section III.B; the graph model is inherited from unpublished reference [9] and is assumed to capture all relevant policy semantics.
domain assumption Node2vec embeddings preserve the structural patterns that distinguish violation classes
Section IV; no analysis or experiment verifies that the chosen embeddings retain the discriminative information needed.
domain assumption Synthetic policies derived from Fedora 39 and Ubuntu Server templates are representative of real SELinux deployments
Section IV.B; the paper explicitly says it does not use complete production policies, yet generalizes to real-world deployment in the conclusion.
domain assumption The reported accuracy on the synthetic dataset estimates performance on unseen policies
Section IV.C; no held-out test set from a different distribution, no cross-validation, and no real-policy evaluation.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Machine Learning-Based Security Policy Analysis." pith.science (2026). https://pith.science/paper/374HTUOF

@misc{pith2026250100085,
  author       = {Pith},
  title        = {Pith review of: Machine Learning-Based Security Policy Analysis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/374HTUOF}},
  note         = {Machine review of arXiv:2501.00085}
}

read the original abstract

Security-Enhanced Linux (SELinux) is a robust security mechanism that enforces mandatory access controls (MAC), but its policy language's complexity creates challenges for policy analysis and management. This research investigates the automation of SELinux policy analysis using graph-based techniques combined with machine learning approaches to detect policy anomalies. The study addresses two key questions: Can SELinux policy analysis be automated through graph analysis, and how do different anomaly detection models compare in analyzing SELinux policies? We will be comparing different machine learning models by evaluating their effectiveness in detecting policy violations and anomalies. Our approach utilizes Neo4j for graph representation of policies, with Node2vec transforming these graph structures into meaningful vector embeddings that can be processed by our machine learning models. In our results, the MLP Neural Network consistently demonstrated superior performance across different dataset sizes, achieving 95% accuracy with balanced precision and recall metrics, while both Random Forest and SVM models showed competitive but slightly lower performance in detecting policy violations. This combination of graph-based modeling and machine learning provides a more sophisticated and automated approach to understanding and analyzing complex SELinux policies compared to traditional manual analysis methods.

Figures

Figures reproduced from arXiv: 2501.00085 by the authors.

**Figure 3.** Bar Graph of Initial 10 Violation Class’ Results [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

**Figure 4.** Bar Graph of 16 Violation Class’ Results [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗

Figures from the paper (1 more)

**Figure 5.** Figure 5: Bar Graph of Final Refined 10 Violation Class’ Results [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

15 extracted references · 14 canonical work pages

[9]

Graph-Powered Mining and Refining of SELinux Security Policies,

A. Eaman, P. Jadczyk, and H. Chipman, "Graph-Powered Mining and Refining of SELinux Security Policies," unpublished, 2024

work page 2024
[1]

Quick start to write a custom SELinux policy,

Red Hat, Inc., "Quick start to write a custom SELinux policy," Red Hat Customer Portal. [Online]. Available: https://access.redhat.com/articles/6999267. [Accessed: Aug. 15, 2024]

work page arXiv 2024
[2]

Visualization-based policy analysis for SELinux: Framework and user study,

W. Xu, M. Shehab, and G. J. Ahn, "Visualization-based policy analysis for SELinux: Framework and user study," Int. J. Inf. Secur., vol. 12, no. 3, pp. 155–171, 2013

work page 2013
[3]

What is SELinux?,

Red Hat, Inc., "What is SELinux?," Red Hat Customer Portal. [Online]. Ch 1-13. Available: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/ 8/html/using_selinux/getting-started-with-selinux_using-selinux. [Accessed: Sept. 3, 2024]

work page 2024
[4]

Security-Enhanced Linux for mortals,

"Security-Enhanced Linux for mortals," YouTube. [Online]. Available: https://www.youtube.com/watch?v=_WOKRaM-HI4. [Accessed: Aug. 28, 2024]

work page 2024
[5]

Mayer, K

F. Mayer, K. MacMillan, and D. Caplan, SELinux by Example: Using Security Enhanced Linux. Pearson Education, 2006

work page 2006
[6]

SETools,

SELinux Project, "SETools," SELinux Project Wiki. [Online]. Available: https://github.com/SELinuxProject/setools. [Accessed: Sept. 12, 2024]

work page 2024
[7]

Automated SELinux RBAC Policy Verification Using SMT

S. Dashevskyi, D. Nisi, Y. Oren, "Automated SELinux RBAC Policy Verification Using SMT," arXiv preprint arXiv:2312.04586, 2023

work page Pith review arXiv 2023

Show all 15 references

[8]

node2vec: Scalable Feature Learning for Networks,

A. Grover and J. Leskovec, "node2vec: Scalable Feature Learning for Networks," in Proc. 22nd ACM SIGKDD Int. Conf. Knowledge Discovery Data Mining, 2016, pp. 855-864, doi: 10.1145/2939672.2939754

2016
[10]

Neo4j Graph Database Platform,

Neo4j, Inc., "Neo4j Graph Database Platform," Neo4j Graph Database Platform. [Online]. Available: https://neo4j.com/
[11]

Operating System Security,

T. Jaeger, "Operating System Security," Synthesis Lectures on Information Security, Privacy, and Trust, vol. 4, no. 1, pp. 1–218, 2008

2008
[12]

Runtime verification of Linux Security Modules policies,

D. Efremov and I. Shchepetkov, "Runtime verification of Linux Security Modules policies," arXiv preprint, arXiv:2001.01442, 2020. [Online]. Available: https://arxiv.org/abs/2001.01442

2001 arXiv
[13]

SPLinux: An Information Flow Secure Linux,

P. Vyas, R. Shyamasundar, B. Patil, S. Borse and S. Sen, "SPLinux: An Information Flow Secure Linux," 2021 IEEE Intl Conf on Parallel & Distributed Processing with Applications, Big Data & Cloud Computing, Sustainable Computing & Communications, Social Computing & Networking (...

2021
[14]

Towards a formal model for security policies specification and validation in the selinux system,

G. Zanin and L. V. Mancini, "Towards a formal model for security policies specification and validation in the selinux system," in Proc. 9th ACM Symp. Access Control Models and Technologies (SACMAT '04), 2004, pp. 136-145, doi: 10.1145/990036.990059

2004
[15]

SPRT: Automatically Adjusting SELinux Policy for Vulnerability Mitigation,

H. Wang, A. Yu, L. Xiao, J. Li, and X. Cao, "SPRT: Automatically Adjusting SELinux Policy for Vulnerability Mitigation," in Proceedings of the 29th ACM Symposium on Access Control Models and Technologies (SACMAT), Jun. 2024, pp. 71-82, doi: 10.1145/3649158.3657306

2024

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.

[1] [9]

Graph-Powered Mining and Refining of SELinux Security Policies,

A. Eaman, P. Jadczyk, and H. Chipman, "Graph-Powered Mining and Refining of SELinux Security Policies," unpublished, 2024

work page 2024

[2] [1]

Quick start to write a custom SELinux policy,

Red Hat, Inc., "Quick start to write a custom SELinux policy," Red Hat Customer Portal. [Online]. Available: https://access.redhat.com/articles/6999267. [Accessed: Aug. 15, 2024]

work page arXiv 2024

[3] [2]

Visualization-based policy analysis for SELinux: Framework and user study,

W. Xu, M. Shehab, and G. J. Ahn, "Visualization-based policy analysis for SELinux: Framework and user study," Int. J. Inf. Secur., vol. 12, no. 3, pp. 155–171, 2013

work page 2013

[4] [3]

What is SELinux?,

Red Hat, Inc., "What is SELinux?," Red Hat Customer Portal. [Online]. Ch 1-13. Available: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/ 8/html/using_selinux/getting-started-with-selinux_using-selinux. [Accessed: Sept. 3, 2024]

work page 2024

[5] [4]

Security-Enhanced Linux for mortals,

"Security-Enhanced Linux for mortals," YouTube. [Online]. Available: https://www.youtube.com/watch?v=_WOKRaM-HI4. [Accessed: Aug. 28, 2024]

work page 2024

[6] [5]

Mayer, K

F. Mayer, K. MacMillan, and D. Caplan, SELinux by Example: Using Security Enhanced Linux. Pearson Education, 2006

work page 2006

[7] [6]

SETools,

SELinux Project, "SETools," SELinux Project Wiki. [Online]. Available: https://github.com/SELinuxProject/setools. [Accessed: Sept. 12, 2024]

work page 2024

[8] [7]

Automated SELinux RBAC Policy Verification Using SMT

S. Dashevskyi, D. Nisi, Y. Oren, "Automated SELinux RBAC Policy Verification Using SMT," arXiv preprint arXiv:2312.04586, 2023

work page Pith review arXiv 2023