A Cybersecurity MLPS Large Language Model with Multi-Path Retrieval Fusion

Junyuan Ma; Liang Shen; Qian Li; Yifan Wan; Yining Hu; Yuan Zhang; Zhenyan Qi

REVIEW 4 major objections 4 minor 9 references

This paper proposes a domain-specific LLM for China's Multi-Level Protection Scheme that fuses three retrieval strategies and reports higher compliance-answer scores than a general retrieval-augmented approach.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-07-31 23:00 UTC pith:FMYNGZMN

load-bearing objection The paper's real contribution is a sensible RAG pipeline for Chinese MLPS compliance QA; the evidence for it is a 10-question comparison scored by an unvalidated LLM judge with a rubric that rewards the system's designed behavior. the 4 major comments →

arxiv 2607.24116 v1 pith:FMYNGZMN submitted 2026-07-27 cs.CR

A Cybersecurity MLPS Large Language Model with Multi-Path Retrieval Fusion

Qian Li , Zhenyan Qi , Liang Shen , Yuan Zhang , Yifan Wan , Junyuan Ma , Yining Hu This is my paper

classification cs.CR

keywords CybersecurityMulti-Level Protection SchemeLarge Language ModelsRetrieval-Augmented Generationcompliance analysishybrid retrievalLLM-as-a-judgehierarchical retrieval

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper aims to show that a large language model built for China's Multi-Level Protection Scheme — a dense, hierarchical body of cybersecurity standards — answers compliance questions more completely and traceably when retrieval is split across three complementary strategies instead of one. It combines hierarchical retrieval by security level, tree-based retrieval over document structure, and token-matching retrieval, then fuses and reranks the candidates. On ten representative questions, the framework scored 6.7 on average versus 6.0 for a general RAG baseline, an 11.7% improvement, winning six of ten questions. The authors argue this provides a more controllable, standards-grounded path for MLPS compliance analysis than manual interpretation or rule-based tools.

Core claim

The central claim is that the multi-path retrieval fusion framework yields higher-quality, more standard-compliant answers for MLPS question answering than a general RAG approach. The authors attribute the gain to combining three retrieval views — exact token matching, structural tree-based retrieval, and security-level hierarchical retrieval — and reranking the merged candidates with source-priority constraints, so that answers cover more control points and cite the relevant clauses. They report an average score of 6.7/10 versus 6.0/10 for the general RAG baseline on ten typical questions, with the biggest wins on questions requiring differentiation of assessment objects and structured 'con

What carries the argument

The load-bearing mechanism is the multi-path retrieval fusion module. It runs three retrievers in parallel: a sparse token-matching retriever that is sensitive to clause numbers and standard names; a tree-based retriever that builds multi-level summaries of document structure so both evidence leaves and section-level summaries are returned; and a hierarchical retriever that first selects relevant documents by security level and then retrieves within them. A unified semantic reranker deduplicates and re-ranks the merged candidates, while file-priority constraints (national standards first) reduce cross-source conflicts. This combination is what the paper claims maintains recall coverage while

Load-bearing premise

The evaluation assumes that an LLM judge's weighted rubric — 5 points for completeness, 3 for accuracy, 2 for traceability — is a valid measure of answer quality, even though traceability explicitly rewards citing the national baseline standard, which is exactly what the proposed framework is engineered to do, and no human-scorer validation is provided.

What would settle it

Give the same ten questions to human MLPS assessors and blind-score the two systems' outputs on the same three dimensions. If human scores show no significant difference between the proposed framework and general RAG — or if the general RAG's poorer traceability does not correspond to lower judged quality — then the reported 0.7-point advantage is an artifact of the LLM judge's rubric rather than a real quality gain.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

If the framework's reported gains hold, MLPS compliance analysis can shift from manual standard interpretation toward LLM-assisted triage that cites specific clauses.
The three retrievers are complementary: losing any one of them should reduce either recall or precision, implying that retrieval fusion, not model size, drives the improvement.
The framework produces answers in a 'conclusion–analysis–basis' structure that matches assessment report conventions, making outputs more deployable in real assessments.
The hierarchical query-routing step (factual vs scenario-based, MLPS vs general cybersecurity vs unrelated) lets the system send only domain queries to the RAG pipeline, reducing irrelevant retrieval.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

A natural testable extension: swap the LLM judge for expert human scoring on the same ten questions; if the 0.7-point gap persists, the result is not an artifact of the rubric.
The multi-path fusion idea generalizes to other dense regulatory domains (e.g., data-protection rules, financial compliance), where exact clause numbers and hierarchical structure matter.
Because the paper reports only ten questions, the average 6.7 vs 6.0 is sensitive to single questions; re-running on a larger balanced sample would tighten or overturn the conclusion.
The framework's failure on question 7 (tool-based testing) suggests coverage of long-tail knowledge is the next bottleneck; fusing a fourth retriever specialized for operational examples could close that gap.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

The paper's real contribution is a sensible RAG pipeline for Chinese MLPS compliance QA; the evidence for it is a 10-question comparison scored by an unvalidated LLM judge with a rubric that rewards the system's designed behavior.

read the letter

Colleague,

The paper is a systems-engineering contribution, not a research breakthrough. It combines three known retrieval strategies—BM25, RAPTOR, and hierarchical clause retrieval—into a single pipeline for Chinese MLPS (Multi-Level Protection Scheme) compliance question answering. That specific combination for this domain is new, and the framework is described clearly enough to reproduce. The authors also do a decent job of motivating the problem: MLPS standards are hierarchical, terminology-dense, and require traceable answers, so a multi-path retrieval design is genuinely sensible.

What it does well: the pipeline logic is sound, the query classification and normalization steps are practical, and the discussion of each retriever's strengths and weaknesses is honest. The knowledge base construction from 39 documents is plausible, and the authors include both positive and negative case analyses, which suggests they are not cherry-picking.

The soft spot is the evaluation, and it is load-bearing. The only quantitative evidence is Table 3: ten questions, average 6.7 versus 6.0 for "general RAG." That baseline is never specified—which retriever, which prompt, which settings? The LLM-as-judge rubric gives 20% weight to traceability, explicitly rewarding citation of GB/T standards, exactly what the framework is engineered to do. The judge is neither validated against human experts nor checked for inter-annotator reliability. With n=10, no error bars, and no significance test, the 0.7-point gap and the 6-wins-2-ties-2-losses tally are well within sampling noise. The stress-test note is right: this is a confounded metric, not evidence of improved answer quality.

There are lesser issues: the title says "Large Language Model" but no model is trained; it's a RAG pipeline on top of existing LLMs. That is a wording problem rather than a deep flaw. The authors also call it a "domain-specific LLM" when they mean "domain-specific framework." Minor.

The central argument—that this pipeline improves MLPS compliance answers—does not hold as currently supported. But the framework concept still deserves scrutiny from someone who knows both RAG and MLPS. A properly blinded human expert evaluation on a larger question set could settle it. I would not desk-reject this; it just needs a more rigorous evaluation before publication.

Recommendation: send to peer review, but with the clear expectation of a major revision focused on the evaluation methodology. The architecture and domain motivation are worth the referee time.

Referee Report

4 major / 4 minor

Summary. The paper proposes a Retrieval-Augmented Generation framework for Chinese Multi-Level Protection Scheme (MLPS) compliance question answering. The framework combines hierarchical retrieval by security level, tree-based retrieval inspired by RAPTOR, and BM25 token matching, plus query classification and rewriting. The empirical section evaluates the framework against a poorly specified 'general RAG' baseline on ten questions using an LLM-as-a-judge rubric that scores completeness (5), accuracy (3), and traceability (2). The authors report an average score of 6.7 for their framework versus 6.0 for the baseline, an 11.7% improvement, and conclude that the multi-path retrieval fusion improves answer quality.

Significance. If the claimed improvement were established, the architectural idea would be relevant to standards-intensive compliance QA: combining exact matching, hierarchical structure, and semantic tree retrieval is a plausible way to balance recall and precision. The paper also covers a practical domain and honestly reports two cases where the baseline outperforms the framework. However, the evidence for the headline claim is not sufficient. The evaluation rests on ten questions, no statistical testing, an unvalidated LLM judge, and a rubric whose traceability component explicitly rewards the framework's designed output. No ablation isolates the contribution of the three retrieval paths, and no code or data are released. The central empirical claim, therefore, is currently unsupported.

major comments (4)

[§4.3, Table 3] The entire empirical support for the central claim is ten questions with integer scores. No standard deviation, confidence interval, or paired significance test is reported. For the eight non-tied items the proposed framework wins 6 and loses 2; a two-sided sign test gives p ≈ 0.145, so the 6.7 vs 6.0 average gap is within sampling noise. As written, the 11.7% improvement claim is not statistically established. A larger, pre-registered question set, or at minimum a paired test with effect sizes, is required.
[§4.2, Table 2] The evaluation metric is confounded with the treatment. The traceability dimension (20% of the score) explicitly rewards references to standards such as GB/T 22239-2019, which is exactly the output the proposed framework is engineered to produce through its multi-path retrieval and 'conclusion–analysis–basis' formatting. No validation of the LLM judge against human domain experts, no inter-annotator agreement, and no blinding are reported. Consequently, the observed advantage may be an artifact of the rubric rather than evidence of better answers. The authors should score answers with human experts blind to system identity or validate the judge against such scores.
[§4.3] The 'general RAG' baseline is underspecified: no retriever, embedding model, chunk size, top-k, prompt template, or generation model are given. Without a controlled baseline, the comparison cannot isolate the effect of multi-path retrieval fusion. The authors should report the baseline configuration and ideally run the same generation model with BM25-only, RAPTOR-only, and hierarchical-only variants.
[§3.2–§4.3] No ablation study is presented for the three retrieval paths. Since the framework's claimed contribution is the fusion of hierarchical, tree-based, and BM25 retrieval, the paper should show that each path contributes and that fusion improves over each individual path. The current end-to-end comparison cannot attribute the score differences to any specific component.

minor comments (4)

[§4.2 vs §4.3] The third dimension is called 'traceability of standards' in §4.2 and 'compliance basis' in §4.3; use consistent terminology.
[Table 3] Only aggregate scores are reported. Per-dimension scores would help assess whether any advantage is concentrated in the traceability dimension, which is precisely the confounded component.
[§4.3.1] The text says 'in Question 2 regarding the differences and relationships between security management assessment and security technical assessment,' but Table 1 lists that as Question 3. The case analysis in §4.3.2 correctly refers to Question 3; the earlier reference appears to be a mislabel.
[§3.1] The base models 'openPangu 7B Model' and 'Qwen' are not specified with versions or configuration details, which limits reproducibility.

Circularity Check

1 steps flagged

The reported 6.7 vs 6.0 advantage rests partly on a rubric that awards 20% of the score for citing MLPS standards—the exact behavior the proposed framework is engineered to produce—and the LLM judge is not validated against human experts.

specific steps

self definitional [Section 4.2 (Table 2) and Section 4.3; cf. Section 1.2 and Section 3.2.4]
"Traceability focuses on whether the response correctly references relevant MLPS standards and supporting clauses, such as GB/T 22239-2019 and GB/T 25070-2019, and whether the cited evidence properly supports the final conclusions. ... The response structure of the proposed framework is more consistent with the writing conventions commonly adopted in classified protection assessment scenarios. Most responses follow a 'conclusion–analysis–basis' organizational pattern, with clear logical structure and better coverage of key points in the questions."

The overall 10-point score used to establish superiority includes a 2-point 'Traceability' term that explicitly rewards citing MLPS standards. The framework is designed to produce exactly this behavior via multi-path retrieval, standard-clause mapping, and the 'conclusion–analysis–basis' output pattern. Thus a portion of the measured score difference (6.7 vs 6.0) is effectively generated by the rubric measuring the system's intended output rather than independently assessed answer quality. No human-expert calibration, inter-annotator agreement, or blinding is reported, so the size of the circular component is not quantified.

full rationale

The paper's architectural derivation is largely self-contained: it combines known retrieval methods (BM25, RAPTOR, hierarchical retrieval) and does not fit parameters to the evaluation data, so there is no fitted-input-called-prediction chain. The central circularity is confined to the evaluation design. The 'traceability' dimension of the rubric is defined as correctly referencing MLPS standards, which is precisely what the proposed multi-path retrieval framework is built to do, while the general RAG baseline is not given the same structural formatting. Since the LLM-as-a-Judge rubric is not validated against human expert scoring, the claimed 11.7% improvement is partly self-referential: the metric rewards the treatment's designed output. This does not make the whole paper circular—completeness (50%) and accuracy (30%) still carry independent content, and some questions show the proposed framework losing—but it substantially weakens the load-bearing quantitative claim. No other significant circularity, such as a load-bearing self-citation chain or a uniqueness argument imported from the authors' prior work, was found.

Axiom & Free-Parameter Ledger

3 free parameters · 6 axioms · 0 invented entities

The paper introduces no new model weights or entities; it assembles existing components (BM25, RAPTOR, openPangu 7B, Qwen) into a pipeline. The hand-set parameters are the evaluation weights and the 10-question test set, which materially shape the reported result.

free parameters (3)

Evaluation rubric weights = completeness=5, accuracy=3, traceability=2
Set by authors in §4.2; the 20% traceability weight is aligned with the framework's purpose-built standard-citation behavior.
Test question set = 10 questions
A hand-picked sample chosen with user organizations (§4); no sampling frame, so scores are not generalizable.
Candidate allocation scheme
The fusion module dynamically allocates candidates per document (§3.2.1), but the exact scheme is not specified, making replication impossible.

axioms (6)

domain assumption The 39-document knowledge base is authoritative and sufficient for MLPS QA
Section 4.1: the knowledge base is the sole evidence source; no external validation of coverage or correctness.
domain assumption LLM-as-a-Judge scores correlate with true answer quality
Section 4.2: the LLM judge is not validated against human expert ratings.
domain assumption The 10 test questions are representative of MLPS QA tasks
Section 4: 'in collaboration with relevant user organizations'; selection criteria are not given.
domain assumption The reference answers are correct and complete
Section 4.2: reference answers are used as ground truth, with no provenance or expert audit.
ad hoc to paper Traceability to standard clauses is a valid quality dimension worth 20% weight
Section 4.2, Table 2: weight chosen by authors; it rewards exactly the framework's configured behavior.
domain assumption openPangu 7B query classification does not introduce errors that materially affect end-to-end scores
Section 4.3.1: classification accuracy is asserted via examples, not measured.

pith-pipeline@v1.3.0-alltime-deepseek · 9401 in / 10407 out tokens · 81355 ms · 2026-07-31T23:00:11.334827+00:00 · methodology

0 comments

read the original abstract

The Multi-Level Protection Scheme (MLPS) is a foundational system in China's cybersecurity governance framework. Therefore, accurate analysis and understanding of MLPS requirements are essential. At present, MLPS analysis still relies mainly on manual interpretation of standards and rule-based tools. This makes it hard to provide stable and consistent compliance analysis in complex application scenarios. The rise of large language models has created new opportunities for making MLPS work more intelligent. However, in standards-intensive and security-sensitive scenarios, general-purpose large language models often cannot ensure controllable reasoning or complete understanding of rules. This paper proposes a large language model framework for MLPS that integrates multiple retrieval strategies. It combines hierarchical retrieval, tree-based retrieval, and tokenization-based matching retrieval. This design helps maintain retrieval coverage while reducing the interference of irrelevant context in the reasoning process. To address the requirements of MLPS question answering for clause accuracy, conclusion traceability, and practical deployability, this paper adopts a evaluation method based on multi-dimensional weighted scoring to quantitatively assess model responses. In comparative experiments on ten typical questions, the proposed domain-specific large language model for MLPS achieved higher overall scores.

discussion (0)

Reference graph

Works this paper leans on

9 extracted references · 2 canonical work pages

[3]

arXiv preprint arXiv:2505.22375

Pangu Embedded: An Efficient Dual-system LLM Reasoner with Metacognition. arXiv preprint arXiv:2505.22375. Della Penna, S., Natella, R., Orbinato, V., et al.,

Pith/arXiv arXiv
[4]

IEEE, pp

Cti -hal: A human -annotated dataset for cyber threat intelligence analysis, 2025 IEEE European Symposium on Security and Privacy Workshops (EuroS&PW). IEEE, pp. 69-78. Divakaran, D.M., Peddinti, S.T.,

2025
[5]

IEEE Security & Privacy 23(5), 38 -45, 10.1109/MSEC.2024.3504512

Large Language Models for Cybersecurity: New Opportunities. IEEE Security & Privacy 23(5), 38 -45, 10.1109/MSEC.2024.3504512. Hu, Y., Zou, F., Han, J., et al.,

arXiv 2024
[6]

Computers & Security 145, 103999, https://doi.org/10.1016/j.cose.2024.103999

LLM-TIKG: Threat intelligence knowledge graph construction utilizing large language model. Computers & Security 145, 103999, https://doi.org/10.1016/j.cose.2024.103999. Huang, H., Sun, N., Tani, M., et al.,

arXiv 2024
[9]

Cybersecurity 8(1), 55, 10.1186/s42400-025-00361-w

When LLMs meet cybersecurity: a systematic literature review. Cybersecurity 8(1), 55, 10.1186/s42400-025-00361-w. Zhang, Z.,

doi:10.1186/s42400-025-00361-w
[2019]

Netinfo Security 19(2), 77-84

Baseline for Classified Protection of Cy bersecurity (GB/T 22239-2019) Standard Interpretation. Netinfo Security 19(2), 77-84. National Information Security Standardization Technical Committee (SAC/TC 260), 2019a. Information security technology —Baseline for classified protection of cybersecurity. State Administration for Market Regulation; Standardizati...

2019
[2021]

Procedia Computer Science 183, 87-93, https://doi.org/10.1016/j.procs.2021.02.034

Compliance analysis of business information system under classified protection 2.0 of cybersecurity. Procedia Computer Science 183, 87-93, https://doi.org/10.1016/j.procs.2021.02.034. Lewis, P., Perez, E., Piktus, A., et al.,

work page doi:10.1016/j.procs.2021.02.034 2021
[2024]

Enhancing IT Security with LLM -Powered Predictive Threat Intelligence, 2024 5th Interna tional Conference on Smart Electronics and 19 Communication (ICOSEC), pp. 751-756. Chen, H., Wang, Y., Han, K., et al.,

2024
[2025]

SHIFRA 2025, 81-97, 10.70470/SHIFRA/2025/005

Beyond Detection: Large Language Models and Next - Generation Cybersecurity. SHIFRA 2025, 81-97, 10.70470/SHIFRA/2025/005. Bokkena, B.,

work page doi:10.70470/shifra/2025/005 2025

[1] [3]

arXiv preprint arXiv:2505.22375

Pangu Embedded: An Efficient Dual-system LLM Reasoner with Metacognition. arXiv preprint arXiv:2505.22375. Della Penna, S., Natella, R., Orbinato, V., et al.,

Pith/arXiv arXiv

[2] [4]

IEEE, pp

Cti -hal: A human -annotated dataset for cyber threat intelligence analysis, 2025 IEEE European Symposium on Security and Privacy Workshops (EuroS&PW). IEEE, pp. 69-78. Divakaran, D.M., Peddinti, S.T.,

2025

[3] [5]

IEEE Security & Privacy 23(5), 38 -45, 10.1109/MSEC.2024.3504512

Large Language Models for Cybersecurity: New Opportunities. IEEE Security & Privacy 23(5), 38 -45, 10.1109/MSEC.2024.3504512. Hu, Y., Zou, F., Han, J., et al.,

arXiv 2024

[4] [6]

Computers & Security 145, 103999, https://doi.org/10.1016/j.cose.2024.103999

LLM-TIKG: Threat intelligence knowledge graph construction utilizing large language model. Computers & Security 145, 103999, https://doi.org/10.1016/j.cose.2024.103999. Huang, H., Sun, N., Tani, M., et al.,

arXiv 2024

[5] [9]

Cybersecurity 8(1), 55, 10.1186/s42400-025-00361-w

When LLMs meet cybersecurity: a systematic literature review. Cybersecurity 8(1), 55, 10.1186/s42400-025-00361-w. Zhang, Z.,

doi:10.1186/s42400-025-00361-w

[6] [2019]

Netinfo Security 19(2), 77-84

Baseline for Classified Protection of Cy bersecurity (GB/T 22239-2019) Standard Interpretation. Netinfo Security 19(2), 77-84. National Information Security Standardization Technical Committee (SAC/TC 260), 2019a. Information security technology —Baseline for classified protection of cybersecurity. State Administration for Market Regulation; Standardizati...

2019

[7] [2021]

Procedia Computer Science 183, 87-93, https://doi.org/10.1016/j.procs.2021.02.034

Compliance analysis of business information system under classified protection 2.0 of cybersecurity. Procedia Computer Science 183, 87-93, https://doi.org/10.1016/j.procs.2021.02.034. Lewis, P., Perez, E., Piktus, A., et al.,

work page doi:10.1016/j.procs.2021.02.034 2021

[8] [2024]

Enhancing IT Security with LLM -Powered Predictive Threat Intelligence, 2024 5th Interna tional Conference on Smart Electronics and 19 Communication (ICOSEC), pp. 751-756. Chen, H., Wang, Y., Han, K., et al.,

2024

[9] [2025]

SHIFRA 2025, 81-97, 10.70470/SHIFRA/2025/005

Beyond Detection: Large Language Models and Next - Generation Cybersecurity. SHIFRA 2025, 81-97, 10.70470/SHIFRA/2025/005. Bokkena, B.,

work page doi:10.70470/shifra/2025/005 2025