{"id":"e8fca32c-37fd-4a83-88e7-3474d7b88733","arxiv_id":"2501.00085","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"A graph-based ML pipeline reports 95% accuracy detecting synthetic SELinux policy violations, but post-hoc violation-class refinement and no real-policy evaluation weaken the central claim.","lead":"This paper turns SELinux security policies into graphs and uses machine learning to flag policy violations, reporting 95% accuracy for a neural network on a small synthetic dataset. The result is plausible as a proof-of-concept, but the evaluation's post-hoc class refinement and lack of real-policy testing undercut the broader claim of automated, production-ready analysis.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 95% accuracy is a selection artifact: the 10-class taxonomy and 455-rule dataset were co-designed using model performance on the same data, so real-world generalization is untested.","rationale":"The reader's weakest assumption — that the reported accuracy is an unbiased estimate of real-world performance — is exactly the load-bearing point, and the manuscript text confirms it is false. The iterative refinement of violation classes is not a minor methodological footnote; it is the mechanism by which the final taxonomy was selected, and it was applied to the same 455-rule dataset that produced the headline Table 5 numbers. This makes the 95% accuracy a measure of fit to a co-adapted labeling scheme rather than evidence that the approach 'significantly outperform[s] existing analysis techniques.' A controlled synthetic study can be a legitimate first step, and the graph construction plus Node2vec pipeline is a plausible proof of concept, but the paper makes a stronger generalization claim. The absence of a real-policy benchmark or even a described train/test split leaves the central claim unsupported. Because the reader already rejected the paper on this basis, my independent review does not move the verdict: it remains REJECT. The proposed concrete test — a frozen model and taxonomy evaluated on independently labeled real policies — would settle whether the concern lands, and if it passed, would substantially strengthen the paper.","tokens_in":10692,"tokens_out":3203,"duration_ms":47874,"concrete_test":"Freeze the final 10-class model and taxonomy exactly as in Table 5, then evaluate it on a held-out corpus of real SELinux policies (e.g., from Fedora 39, RHEL 9, or Ubuntu Server) whose violations are labeled independently by SELinux experts, with no class redefinition and no retraining after seeing results. If the macro F1 on this real-policy test falls materially below the reported 0.95 (e.g., below 0.85), the central automation/generalization claim fails; if it remains near 0.95, the concern is answered.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central 95% accuracy claim (Abstract, Section IV.C, Table 5) is not an unbiased estimate of detection performance, because the label taxonomy and the dataset were co-adapted to the models on the same 455 rules. Section IV.C documents an iterative process: the initial 5 classes were split because of false positives; 10 classes were expanded to 16 when SoD/Contradictory/Missing classes underperformed; then 16 were consolidated to the final 10 because models showed overlap and low recall. The text says the refinement 'involved strategic consolidation of related violation types' and that consolidation 'was driven by our observation that the models struggled with overly specific access violations.' Thus the final class boundaries were chosen after seeing model errors on the exact dataset used to report Table 5. The reported 0.93-0.95 accuracies therefore measure how well the models fit a taxonomy optimized to their mistakes on that same data; they do not measure detection of real SELinux misconfigurations. No independent held-out set, no real-policy test, no described cross-validation procedure, and no comparison with SETools, SPRT, or formal-analysis tools is reported, so the abstract's 'significantly outperforming existing analysis techniques' is unsupported. The conclusion's claim of up to 96% accuracy inherits the same problem.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a graph-based machine learning pipeline for detecting violations in SELinux policies. Policies are represented as Neo4j graphs, embedded with Node2vec, and classified by Random Forest, SVM, and MLP models. On a synthetic dataset of 455 policy rules, the MLP reaches 95% accuracy with macro precision/recall/F1 of 0.95/0.97/0.95, and the authors conclude that the approach automates SELinux policy analysis and significantly outperforms existing techniques. The study also reports experiments with 5, 10, and 16 violation classes, documenting how the class schema was iteratively refined.","tokens_in":10950,"tokens_out":3552,"duration_ms":35530,"significance":"If the reported results were reliable, the paper would be a useful step toward automated SELinux policy auditing, and the comparison of three classifiers on graph embeddings is a reasonable research question. The authors deserve credit for building a concrete pipeline (Neo4j + Node2vec + ML classifiers) and for being unusually transparent about the iterative refinement of their violation taxonomy. However, the central accuracy claim is undermined by a fundamental methodological flaw: the violation classes were adjusted based on model performance on the same dataset used to report the final accuracy, and no held-out or real-policy evaluation is provided. The dependence on an unpublished self-cited manuscript for the graph model, and the absence of code, data, and baseline comparisons, further reduce the contribution to an unvalidated proof-of-concept.","major_comments":[{"comment":"The reported accuracies (including the abstract's 95% MLP figure) are not unbiased estimates of detection performance, because the violation-class taxonomy was iteratively adjusted based on model performance on the same 455-rule dataset. The text documents that the initial 5 classes were split due to high false positives, that classes were expanded to 16 when SoD/Contradictory/Missing rules underperformed, and that classes were then consolidated to 10 because the 'models struggled with overly specific access violations' and showed low recall for missing-rule classes (Table 4). Thus the final Table 5 numbers measure how well the models fit a taxonomy optimized to their errors on the same data, not the detection of real SELinux misconfigurations.","section":"Section IV.C, Tables 2–5"},{"comment":"The evaluation is performed solely on a self-constructed synthetic dataset of 455 policy rules derived from Fedora 39 and Ubuntu Server templates. The paper does not describe any train/test split, cross-validation procedure, or held-out set, so the reported accuracies could reflect memorization rather than generalization. In addition, no comparison is made with SETools, SPRT, SMT-based verification, or formal-analysis tools, leaving the abstract's claim that the method 'significantly outperform[s] existing analysis techniques' without empirical support.","section":"Section IV.B"},{"comment":"The per-class recall for 'Missing necessary file access for system processes' is 0.25 for Random Forest and MLP and 0.06 for SVM. The paper's narrative states that this observation drove the consolidation of four separate access-violation classes into a single class. This is direct evidence that the final 10-class schema was chosen in response to the models' weaknesses on the very data used to report the final accuracy, which constitutes selection on the test set and invalidates the headline numbers.","section":"Section IV.C, Table 4"},{"comment":"The graph model is based on an unpublished, self-cited manuscript by Eaman et al. [9], so the reader cannot verify the representation or assess its novelty relative to prior work. The paper also does not report the Node2vec hyperparameters, the MLP architecture, or the training procedure, and no source code or synthetic dataset is provided. These omissions are load-bearing because the entire pipeline depends on those embedding and training choices, making the reported results irreproducible.","section":"Section III.B and reference [9]"}],"minor_comments":[{"comment":"Listing 5 is introduced as 'ten distinct violation classes' but shows eleven numbered entries (0 through 10), and the mapping from these numbers to the classes discussed in the text is confusing; clarify the numbering and reconcile it with the subsequent tables.","section":"Section IV.A"},{"comment":"There is a typographical error: 'a balanced amount of each violatioin class' should read 'violation class'.","section":"Section IV.C"},{"comment":"The tables render with broken lines inside numeric values (e.g., '0.7 9' and '0.8 5'), which appears to be a formatting defect; fix the table layout so that each metric is readable.","section":"Tables 1–5"},{"comment":"The phrase 'significantly outperforming existing analysis techniques' is used in the abstract and introduction, but no comparative experiments against existing tools are presented anywhere in the manuscript; either add such a comparison or weaken the claim.","section":"Abstract and Section I"},{"comment":"Reference [4] is a YouTube video, which is an unconventional citation for a technical claim about SELinux; consider replacing it with a primary source. Also, the reference list order and formatting are inconsistent with a standard IEEE style.","section":"References"}],"recommendation":"reject","confidential_remarks":"The paper appears to be a workshop-level student project. The central accuracy claim is not credible as presented because of the post-hoc class refinement and the absence of any held-out evaluation; fixing this requires a substantially new experimental design, which is beyond the scope of a revision. I would also note that the approach leans heavily on an unpublished self-citation [9], and the lack of code/data makes it impossible to verify the pipeline. If the authors return with a pre-registered taxonomy, a real-policy test set, and comparisons with existing tools, the idea could be worth revisiting."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe headline number is not trustworthy, but the paper is not a throwaway. The authors build a graph representation of SELinux policy rules, embed it with Node2vec, and run RF/SVM/MLP classifiers on a self-constructed synthetic dataset of 455 rules. That specific combination is new in the SELinux tooling space and it is a reasonable proof-of-concept. They also deserve credit for documenting the iterative taxonomy refinement in Section IV.C: starting with 5 classes, splitting and merging based on model errors, ending at 10. That transparency is what lets a reader see the central problem.\n\nThe soft spot is load-bearing. The 95% accuracy reported in Table 5 is a selection artifact. The final 10-class schema was chosen after observing which classes the models confused—consolidating \"overly permissive\" into \"improper privilege,\" merging contradictory-rule subclasses, and grouping four access-missing classes because recall was 0.25. The same 455-rule dataset was then used to report the final accuracy. There is no described train/test split, no cross-validation, and no independent held-out set. So the numbers measure how well the models fit a taxonomy that was tuned to their mistakes on that exact data. Real-policy generalization is untested. The abstract's claim of \"significantly outperforming existing analysis techniques\" is also unsupported: no SETools, SPRT, or other baselines were run.\n\nThe reliance on an unpublished, self-cited manuscript [9] for the graph model is a secondary concern. It may be fine, but as submitted it is not independently checkable.\n\nNone of this means the direction is wrong. The violation taxonomy, while repackaging known SELinux issues, is sensible, and the idea of learning embeddings over policy graphs is plausible. The paper is an honest proof-of-concept that needs a rigorous evaluation: held-out policies, real production policies, and comparison against existing tools.\n\nMy recommendation: send it to peer review rather than desk reject. A serious referee would demand that evaluation, and the authors seem capable of providing it. As submitted, the central claim does not hold.","headline":"The paper is an honest but flawed proof-of-concept: the headline 95% accuracy is a selection artifact from co-adapting the class taxonomy to model errors on the same dataset.","tokens_in":11474,"tokens_out":2349,"would_cite":false,"duration_ms":24303,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Graph-based machine learning automates SELinux policy analysis, with an MLP neural network reaching 95% accuracy.","keywords":["SELinux","policy analysis","mandatory access control","graph-based machine learning","Node2vec","anomaly detection","neural network","security policy"],"falsifier":"Train the pipeline on the paper's synthetic set, then run it on a real production SELinux policy whose violations have been labeled by independent SELinux experts; if accuracy on that held-out policy falls to near chance levels, the reported generalization claim fails. A cheaper check is to retrain with the 16-class scheme while holding out entire violation classes during training and measuring recall on those held-out classes.","tokens_in":10496,"feed_emoji":"🛡️","tokens_out":4573,"duration_ms":42376,"temperature":0.7,"pith_summary":"Security-Enhanced Linux (SELinux) enforces mandatory access controls through a policy language so intricate that manual analysis is error-prone and existing tools produce hard-to-interpret abstractions. This paper tries to establish that policy analysis can be automated by converting SELinux type-enforcement rules into a graph, embedding the graph with Node2vec, and feeding those embeddings into standard classifiers. The authors report that a Multi-Layer Perceptron (MLP) neural network detects policy violations with 95% accuracy, balanced precision/recall of 0.95/0.97/0.95, outperforming Random Forest (93%) and SVM (92%). If the result holds, administrators could audit SELinux policies automatically and catch misconfigurations such as separation-of-duty breaks and domain-transition issues without combing through thousands of rules by hand.","feed_headline":"Neural net catches SELinux policy flaws at 95% accuracy","feed_subtitle":"Graph embeddings of security policies let an MLP outscore Random Forest and SVM on violation detection.","key_machinery":"The central machinery is the graph model of SELinux type enforcement: types become nodes and allow rules become edges, stored in the Neo4j graph database. Node2vec generates continuous vector embeddings that preserve local and global graph structure through flexible random walks, and these embeddings are the input to Random Forest, SVM, and MLP classifiers. The paper also relies on a refined taxonomy of ten violation classes, consolidated from an initial sixteen, which it argues balances specificity with generalization for detection.","core_discovery":"The paper claims that representing SELinux type-enforcement rules as a graph, with subject, object, and class nodes connected by allow-rule edges, and then learning node embeddings via Node2vec, yields feature vectors that preserve policy neighborhoods and violation patterns. On a synthetic dataset of 455 policy rules spanning ten violation classes, the MLP classifier reaches 95% accuracy with a macro F1 of 0.95, the best of the three models tested. The authors further claim that this graph-based approach significantly outperforms existing analysis techniques by capturing relational structure that prior machine-learning attempts missed.","pith_inferences":["The authors' iterative refinement of violation classes based on model performance on the same dataset risks circularity: part of the reported 95% accuracy may reflect classes chosen to be easy for the models rather than intrinsically detectable violations.","A stronger test would freeze the class taxonomy before model development and evaluate on a separate, independently labeled real policy; the paper reports no such held-out test.","The graph representation could transfer to other mandatory-access-control systems, such as AppArmor or Windows integrity levels, where policies also form subject-object-permission relations, though the violation taxonomy would need re-deriving.","Because the dataset is synthetic and server-focused, the next validation step implied by the paper's own future-work section is measuring performance on production policies with hundreds of thousands of rules."],"forward_implications":["Automated policy auditing becomes feasible: the graph-embedding pipeline can flag candidate violations in policies with thousands of rules, reducing the manual inspection burden on administrators.","The MLP's consistent edge over Random Forest and SVM across dataset sizes (125, 401, and 455 rules) indicates that the neural net benefits most from the graph-derived embeddings.","The trajectory from 16 classes down to 10 shows that overly fine-grained violation categories hurt detection of broader patterns, making the consolidated taxonomy part of the claimed performance.","The same graph-embedding pipeline is claimed to generalize across policy aspects, including transition graphs, attribute graphs, and object-class graphs, suggesting one framework can cover multiple SELinux rule types."],"supporting_citations":[{"why":"It supplies the node2vec embedding algorithm that turns policy graphs into feature vectors.","marker":"[8]"},{"why":"It provides the graph model with Subject, Object, and Class nodes that the paper extends for SELinux policy representation.","marker":"[9]"},{"why":"It grounds the policy-language syntax and the common violation types on which the classes are based.","marker":"[3]"},{"why":"It is the reference for SELinux type enforcement and security-context structure.","marker":"[5]"},{"why":"It is the graph database used to store and query the policy graphs.","marker":"[10]"},{"why":"It motivates the relational graph-based representation and its scalability for security-policy analysis.","marker":"[2]"},{"why":"It represents the SMT-based formal verification approach that the paper claims to outperform in usability.","marker":"[7]"},{"why":"It is the alternative ML-based SELinux policy tool, SPRT, that frames the comparison of automated vulnerability-mitigation approaches.","marker":"[15]"}],"fun_headline_variants":["Graph ML finds SELinux policy flaws with 95% accuracy","Neural net tops rivals on SELinux anomaly detection","Graph embeddings + MLP automate SELinux policy checks","MLP wins on SELinux policy violation detection"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The 95% accuracy is an unbiased estimate of real-world detection performance, yet the evaluation uses a self-constructed synthetic dataset whose ten violation classes were iteratively redefined based on how well the models scored on that same data, with no held-out real-policy test.","fun_headline_variants_meta":{"raw":{"variants":["Graph ML finds SELinux policy flaws with 95% accuracy","Neural net tops rivals on SELinux anomaly detection","Graph embeddings + MLP automate SELinux policy checks","MLP wins on SELinux policy violation detection"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000664,"raw_usage":{"total_tokens":2981,"prompt_tokens":846,"completion_tokens":2135,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":462,"completion_tokens_details":{"reasoning_tokens":2071}},"tokens_in":462,"tokens_out":2135,"duration_ms":17281,"temperature":1.0,"reasoning_tokens":2071,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T23:00:27.227864+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the pipeline on the paper's synthetic set, then run it on a real production SELinux policy whose violations have been labeled by independent SELinux experts; if accuracy on that held-out policy falls to near chance levels, the reported generalization claim fails. A cheaper check is to retrain with the 16-class scheme while holding out entire violation classes during training and measuring recall on those held-out classes.","supporting_citations":[{"cited_title":"Graph-Powered Mining and Refining of SELinux Security Policies,","cited_arxiv_id":null,"evidence_quote":"It provides the graph model with Subject, Object, and Class nodes that the paper extends for SELinux policy representation."},{"cited_title":"What is SELinux?,","cited_arxiv_id":null,"evidence_quote":"It grounds the policy-language syntax and the common violation types on which the classes are based."},{"cited_title":"Mayer, K","cited_arxiv_id":null,"evidence_quote":"It is the reference for SELinux type enforcement and security-context structure."},{"cited_title":"Neo4j Graph Database Platform,","cited_arxiv_id":null,"evidence_quote":"It is the graph database used to store and query the policy graphs."},{"cited_title":"Visualization-based policy analysis for SELinux: Framework and user study,","cited_arxiv_id":null,"evidence_quote":"It motivates the relational graph-based representation and its scalability for security-policy analysis."},{"cited_title":"Automated SELinux RBAC Policy Verification Using SMT","cited_arxiv_id":"2312.04586","evidence_quote":"It represents the SMT-based formal verification approach that the paper claims to outperform in usability."},{"cited_title":"SPRT: Automatically Adjusting SELinux Policy for Vulnerability Mitigation,","cited_arxiv_id":null,"evidence_quote":"It is the alternative ML-based SELinux policy tool, SPRT, that frames the comparison of automated vulnerability-mitigation approaches."}],"review_version":1}