{"id":"aec234de-27f8-4499-ad83-898dd64939eb","arxiv_id":"2607.24116","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"A RAG pipeline that fuses BM25, RAPTOR, and hierarchical retrieval scores slightly higher than plain RAG on ten MLPS compliance questions, but the evaluation is small and partly circular.","lead":"This paper describes a question-answering system for China's Multi-Level Protection Scheme (MLPS) cybersecurity standards. It combines three retrieval methods (BM25, tree-based RAPTOR, and hierarchical retrieval) around a large language model and reports higher scores than a standard RAG baseline on ten test questions.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported 6.7 vs 6.0 advantage rests on an unvalidated LLM-judge rubric that rewards the framework's designed output; blinded human expert scoring would settle it.","rationale":"The paper's strongest claim is the empirical superiority of the proposed framework, reported as an average score of 6.7 versus 6.0 on ten questions (Section 4.3, Table 3). For that claim to hold, the evaluation metric must be a valid, unbiased measure of answer quality. The rubric in Section 4.2 explicitly rewards traceability to MLPS standards, which is exactly the behavior the proposed retrieval-fusion framework is designed to produce; giving 20% of the score to that behavior creates a built-in advantage independent of actual correctness or usefulness. No evidence is presented that this LLM judge agrees with human expert judgment, that scores are stable across judges or prompts, or that the 'general RAG' baseline was configured comparably. The small sample and the absence of any statistical test further amplify the risk that the observed difference is noise. This is a load-bearing internal-validity concern, not a disagreement with community consensus. The reader's weakest assumption identifies the same issue, so I agree. A blinded human expert evaluation is the concrete, feasible check that would resolve whether the claimed improvement is real or an artifact of the rubric; until that is done, the central claim is not established. The title overclaim and unspecified baseline are additional problems, but the judge-validity threat is the most direct route to testing the headline result.","tokens_in":9582,"tokens_out":3458,"duration_ms":32600,"concrete_test":"Run a double-blind human evaluation: recruit at least three MLPS assessment experts; give them the 20 model outputs (10 proposed, 10 general RAG, randomized and de-identified) plus the same reference answers, and ask them to rate the three rubric dimensions independently. Compare their scores to the LLM judge's scores (e.g., Spearman correlation and the paired-difference distribution between systems). If the human-expert average difference is not close to 0.7 in the same direction, or if the LLM judge and experts disagree substantially, the central claim is unsupported. Also report per-dimension expert scores to test whether the apparent traceability advantage is an artifact of the rubric.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Table 3 is the only quantitative support for the central claim. The comparison's validity depends on the LLM-as-a-judge rubric (Section 4.2) being an unbiased measure of answer quality. It is not shown to be: the traceability/compliance dimension (2/10 points) explicitly awards correctly referencing standards such as GB/T 22239-2019 and GB/T 25070-2019, which is precisely what the proposed framework is engineered to do via multi-path retrieval and 'conclusion–analysis–basis' formatting. Since the same authors designed both the framework and the rubric, the metric is confounded with the treatment. There is no human-expert calibration, no inter-annotator agreement, no blinding, and no analysis of judge sensitivity to phrasing versus content. Additionally, n=10 with 6 wins, 2 ties, 2 losses and no error bars or paired significance test does not rule out chance; a 0.7 average gap on a 10-point scale across 10 items is not shown to be statistically reliable. This matters because if the judge systematically inflates citation-heavy, structured answers, the average-score difference does not establish that the multi-path retrieval fusion improves answer quality.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a Retrieval-Augmented Generation framework for Chinese Multi-Level Protection Scheme (MLPS) compliance question answering. The framework combines hierarchical retrieval by security level, tree-based retrieval inspired by RAPTOR, and BM25 token matching, plus query classification and rewriting. The empirical section evaluates the framework against a poorly specified 'general RAG' baseline on ten questions using an LLM-as-a-judge rubric that scores completeness (5), accuracy (3), and traceability (2). The authors report an average score of 6.7 for their framework versus 6.0 for the baseline, an 11.7% improvement, and conclude that the multi-path retrieval fusion improves answer quality.","tokens_in":9922,"tokens_out":4661,"duration_ms":43944,"significance":"If the claimed improvement were established, the architectural idea would be relevant to standards-intensive compliance QA: combining exact matching, hierarchical structure, and semantic tree retrieval is a plausible way to balance recall and precision. The paper also covers a practical domain and honestly reports two cases where the baseline outperforms the framework. However, the evidence for the headline claim is not sufficient. The evaluation rests on ten questions, no statistical testing, an unvalidated LLM judge, and a rubric whose traceability component explicitly rewards the framework's designed output. No ablation isolates the contribution of the three retrieval paths, and no code or data are released. The central empirical claim, therefore, is currently unsupported.","major_comments":[{"comment":"The entire empirical support for the central claim is ten questions with integer scores. No standard deviation, confidence interval, or paired significance test is reported. For the eight non-tied items the proposed framework wins 6 and loses 2; a two-sided sign test gives p ≈ 0.145, so the 6.7 vs 6.0 average gap is within sampling noise. As written, the 11.7% improvement claim is not statistically established. A larger, pre-registered question set, or at minimum a paired test with effect sizes, is required.","section":"§4.3, Table 3"},{"comment":"The evaluation metric is confounded with the treatment. The traceability dimension (20% of the score) explicitly rewards references to standards such as GB/T 22239-2019, which is exactly the output the proposed framework is engineered to produce through its multi-path retrieval and 'conclusion–analysis–basis' formatting. No validation of the LLM judge against human domain experts, no inter-annotator agreement, and no blinding are reported. Consequently, the observed advantage may be an artifact of the rubric rather than evidence of better answers. The authors should score answers with human experts blind to system identity or validate the judge against such scores.","section":"§4.2, Table 2"},{"comment":"The 'general RAG' baseline is underspecified: no retriever, embedding model, chunk size, top-k, prompt template, or generation model are given. Without a controlled baseline, the comparison cannot isolate the effect of multi-path retrieval fusion. The authors should report the baseline configuration and ideally run the same generation model with BM25-only, RAPTOR-only, and hierarchical-only variants.","section":"§4.3"},{"comment":"No ablation study is presented for the three retrieval paths. Since the framework's claimed contribution is the fusion of hierarchical, tree-based, and BM25 retrieval, the paper should show that each path contributes and that fusion improves over each individual path. The current end-to-end comparison cannot attribute the score differences to any specific component.","section":"§3.2–§4.3"}],"minor_comments":[{"comment":"The third dimension is called 'traceability of standards' in §4.2 and 'compliance basis' in §4.3; use consistent terminology.","section":"§4.2 vs §4.3"},{"comment":"Only aggregate scores are reported. Per-dimension scores would help assess whether any advantage is concentrated in the traceability dimension, which is precisely the confounded component.","section":"Table 3"},{"comment":"The text says 'in Question 2 regarding the differences and relationships between security management assessment and security technical assessment,' but Table 1 lists that as Question 3. The case analysis in §4.3.2 correctly refers to Question 3; the earlier reference appears to be a mislabel.","section":"§4.3.1"},{"comment":"The base models 'openPangu 7B Model' and 'Qwen' are not specified with versions or configuration details, which limits reproducibility.","section":"§3.1"}],"recommendation":"reject","confidential_remarks":"The manuscript reads like an extended abstract or workshop paper. For a journal, the evaluation is far below the bar: the rubric circularity and unvalidated LLM judge would require a substantial new human study, a much larger question set, and a proper ablation. These are not minor local fixes but a fundamental rework of the empirical support. I would not invite a revision without that new evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper is a systems-engineering contribution, not a research breakthrough. It combines three known retrieval strategies—BM25, RAPTOR, and hierarchical clause retrieval—into a single pipeline for Chinese MLPS (Multi-Level Protection Scheme) compliance question answering. That specific combination for this domain is new, and the framework is described clearly enough to reproduce. The authors also do a decent job of motivating the problem: MLPS standards are hierarchical, terminology-dense, and require traceable answers, so a multi-path retrieval design is genuinely sensible.\n\nWhat it does well: the pipeline logic is sound, the query classification and normalization steps are practical, and the discussion of each retriever's strengths and weaknesses is honest. The knowledge base construction from 39 documents is plausible, and the authors include both positive and negative case analyses, which suggests they are not cherry-picking.\n\nThe soft spot is the evaluation, and it is load-bearing. The only quantitative evidence is Table 3: ten questions, average 6.7 versus 6.0 for \"general RAG.\" That baseline is never specified—which retriever, which prompt, which settings? The LLM-as-judge rubric gives 20% weight to traceability, explicitly rewarding citation of GB/T standards, exactly what the framework is engineered to do. The judge is neither validated against human experts nor checked for inter-annotator reliability. With n=10, no error bars, and no significance test, the 0.7-point gap and the 6-wins-2-ties-2-losses tally are well within sampling noise. The stress-test note is right: this is a confounded metric, not evidence of improved answer quality.\n\nThere are lesser issues: the title says \"Large Language Model\" but no model is trained; it's a RAG pipeline on top of existing LLMs. That is a wording problem rather than a deep flaw. The authors also call it a \"domain-specific LLM\" when they mean \"domain-specific framework.\" Minor.\n\nThe central argument—that this pipeline improves MLPS compliance answers—does not hold as currently supported. But the framework concept still deserves scrutiny from someone who knows both RAG and MLPS. A properly blinded human expert evaluation on a larger question set could settle it. I would not desk-reject this; it just needs a more rigorous evaluation before publication.\n\nRecommendation: send to peer review, but with the clear expectation of a major revision focused on the evaluation methodology. The architecture and domain motivation are worth the referee time.","headline":"The paper's real contribution is a sensible RAG pipeline for Chinese MLPS compliance QA; the evidence for it is a 10-question comparison scored by an unvalidated LLM judge with a rubric that rewards the system's designed behavior.","tokens_in":10298,"tokens_out":2048,"would_cite":false,"duration_ms":18165,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper proposes a domain-specific LLM for China's Multi-Level Protection Scheme that fuses three retrieval strategies and reports higher compliance-answer scores than a general retrieval-augmented approach.","keywords":["Cybersecurity","Multi-Level Protection Scheme","Large Language Models","Retrieval-Augmented Generation","compliance analysis","hybrid retrieval","LLM-as-a-judge","hierarchical retrieval"],"falsifier":"Give the same ten questions to human MLPS assessors and blind-score the two systems' outputs on the same three dimensions. If human scores show no significant difference between the proposed framework and general RAG — or if the general RAG's poorer traceability does not correspond to lower judged quality — then the reported 0.7-point advantage is an artifact of the LLM judge's rubric rather than a real quality gain.","tokens_in":9534,"feed_emoji":"🛡️","tokens_out":4192,"duration_ms":33437,"temperature":0.7,"pith_summary":"The paper aims to show that a large language model built for China's Multi-Level Protection Scheme — a dense, hierarchical body of cybersecurity standards — answers compliance questions more completely and traceably when retrieval is split across three complementary strategies instead of one. It combines hierarchical retrieval by security level, tree-based retrieval over document structure, and token-matching retrieval, then fuses and reranks the candidates. On ten representative questions, the framework scored 6.7 on average versus 6.0 for a general RAG baseline, an 11.7% improvement, winning six of ten questions. The authors argue this provides a more controllable, standards-grounded path for MLPS compliance analysis than manual interpretation or rule-based tools.","feed_headline":"Hybrid retrieval lifts MLPS answers 11.7 percent over plain RAG","feed_subtitle":"A domain LLM with three fused retrievers beats general RAG on 6 of 10 compliance questions.","key_machinery":"The load-bearing mechanism is the multi-path retrieval fusion module. It runs three retrievers in parallel: a sparse token-matching retriever that is sensitive to clause numbers and standard names; a tree-based retriever that builds multi-level summaries of document structure so both evidence leaves and section-level summaries are returned; and a hierarchical retriever that first selects relevant documents by security level and then retrieves within them. A unified semantic reranker deduplicates and re-ranks the merged candidates, while file-priority constraints (national standards first) reduce cross-source conflicts. This combination is what the paper claims maintains recall coverage while","core_discovery":"The central claim is that the multi-path retrieval fusion framework yields higher-quality, more standard-compliant answers for MLPS question answering than a general RAG approach. The authors attribute the gain to combining three retrieval views — exact token matching, structural tree-based retrieval, and security-level hierarchical retrieval — and reranking the merged candidates with source-priority constraints, so that answers cover more control points and cite the relevant clauses. They report an average score of 6.7/10 versus 6.0/10 for the general RAG baseline on ten typical questions, with the biggest wins on questions requiring differentiation of assessment objects and structured 'con","pith_inferences":["A natural testable extension: swap the LLM judge for expert human scoring on the same ten questions; if the 0.7-point gap persists, the result is not an artifact of the rubric.","The multi-path fusion idea generalizes to other dense regulatory domains (e.g., data-protection rules, financial compliance), where exact clause numbers and hierarchical structure matter.","Because the paper reports only ten questions, the average 6.7 vs 6.0 is sensitive to single questions; re-running on a larger balanced sample would tighten or overturn the conclusion.","The framework's failure on question 7 (tool-based testing) suggests coverage of long-tail knowledge is the next bottleneck; fusing a fourth retriever specialized for operational examples could close that gap."],"forward_implications":["If the framework's reported gains hold, MLPS compliance analysis can shift from manual standard interpretation toward LLM-assisted triage that cites specific clauses.","The three retrievers are complementary: losing any one of them should reduce either recall or precision, implying that retrieval fusion, not model size, drives the improvement.","The framework produces answers in a 'conclusion–analysis–basis' structure that matches assessment report conventions, making outputs more deployable in real assessments.","The hierarchical query-routing step (factual vs scenario-based, MLPS vs general cybersecurity vs unrelated) lets the system send only domain queries to the RAG pipeline, reducing irrelevant retrieval."],"fun_headline_variants":["Fused multi-path retrieval lifts MLPS QA 11.7%","Three retrievers beat one: MLPS LLM scores 6.7 vs 6.0","Hybrid retrieval fusion improves MLPS compliance answers","Fused retrievers give MLPS LLM edge over plain RAG"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The evaluation assumes that an LLM judge's weighted rubric — 5 points for completeness, 3 for accuracy, 2 for traceability — is a valid measure of answer quality, even though traceability explicitly rewards citing the national baseline standard, which is exactly what the proposed framework is engineered to do, and no human-scorer validation is provided.","fun_headline_variants_meta":{"raw":{"variants":["Fused multi-path retrieval lifts MLPS QA 11.7%","Three retrievers beat one: MLPS LLM scores 6.7 vs 6.0","Hybrid retrieval fusion improves MLPS compliance answers","Fused retrievers give MLPS LLM edge over plain RAG"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00038,"raw_usage":{"total_tokens":1840,"prompt_tokens":718,"completion_tokens":1122,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":462,"completion_tokens_details":{"reasoning_tokens":1054}},"tokens_in":462,"tokens_out":1122,"duration_ms":9865,"temperature":1.0,"reasoning_tokens":1054,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-31T23:00:11.334827+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Give the same ten questions to human MLPS assessors and blind-score the two systems' outputs on the same three dimensions. If human scores show no significant difference between the proposed framework and general RAG — or if the general RAG's poorer traceability does not correspond to lower judged quality — then the reported 0.7-point advantage is an artifact of the LLM judge's rubric rather than a real quality gain.","supporting_citations":[],"review_version":1}