{"id":"b111edcd-a96f-4b3f-a7ef-ddffb7fb353c","arxiv_id":"2507.06497","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"TELSAFE computes numeric security risk scores by multiplying empirical frequencies of CVSS/EPSS attributes in an event tree, demonstrated on a CVE dataset for the telecommunications sector.","lead":"This paper introduces TELSAFE, a hybrid security risk assessment framework that combines qualitative ISO-style steps with event-tree probability calculations to produce numeric risk scores. The authors illustrate it on a public CVE dataset for the telecommunications sector, but provide no validation, no code, and no reproducible numbers.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The computed P[π] is a product of CVSS attribute frequencies, not a probability of a standards-gap incident; the risk score lacks a defined target event.","rationale":"The reader's weakest assumption correctly identifies the unrepresentative proxy dataset, and that is a real problem. My concern is more structural: even if the Kaggle data were perfectly representative of standards-gap vulnerabilities, the computation in Algorithm 1 does not produce a likelihood of any adverse event. It produces the product of marginal frequencies of CVSS metadata values under an independence assumption. The event tree defined in Section III-B describes chains of events with outcomes like alarm triggered or sprinkler activated, but the use case replaces these with static CVSS attributes. No event in the tree corresponds to 'a standards gap leads to compromise' or even 'the vulnerability is exploited.' Consequently, Eq. 10 multiplies a non-likelihood by a CVSS-derived impact to get a number that is labeled a risk score. The paper's own statements support this: Section IV-A says the dataset was not acquired as data for gaps, and Section IV-C.5.c says no labeled ground truth exists for validation. The CISA KEV field is present in the dataset but unused, which provides an immediate empirical check. Because the core quantity P[π] has no clear real-world referent, the framework cannot be considered a quantitative risk assessment without an additional, unprovided link between CVE metadata frequencies and incident likelihood. This reinforces rather than overturns the reader's REJECT verdict, so I recommend UNCHANGED. I only partially agree with the reader because the load-bearing failure is the construct validity of P[π] itself, not just the choice of dataset; the dataset issue is a symptom of the missing target-event definition.","tokens_in":18477,"tokens_out":3850,"duration_ms":45717,"concrete_test":"Using the same Kaggle dataset, reproduce Algorithm 1 to compute Rπ for every CVE, then split CVEs by the CISA KEV flag already present in the dataset (known exploited vs not). Compute AUROC or rank-biserial correlation of Rπ (and of P[π] alone) against KEV status. If the scores do not rank known-exploited vulnerabilities above non-exploited ones (e.g., AUROC not significantly above 0.5), then the claimed likelihood measure does not track real-world exploitation, and the central claim fails. As a secondary check, test the independence assumption: compute the observed joint frequency of each attribute combination and compare it with the product of marginals; large discrepancies indicate Eq. 7 is misapplied.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that TELSAFE computes a quantitative risk score Rπ = P[π] × γπ (Eq. 10) for standards-implementation gaps. The load-bearing step is Algorithm 1's construction of P[π]. For each CVE row, P[π] is the product of empirical marginal frequencies P[ejx] of eight CVSS attributes over the whole Kaggle corpus (Eq. 7, with independence assumed). This quantity is not the probability of any event in the risk scenario: the event tree's basic events are attributes such as attack vector and scope, not 'a standards-gap incident occurs' or 'the vulnerability is exploited.' Even if the Kaggle dataset were a perfect sample of standards-gap vulnerabilities, the product of marginal attribute frequencies would only estimate the probability of observing that attribute combination in the CVE corpus; it would not estimate the likelihood of an incident or a gap-induced compromise. Section IV-A explicitly states the dataset 'has not been acquired by the source as data for gaps between standards and implementation,' and Section IV-C.5.c concedes there is no labeled ground truth. Thus the mapping from P[π] to 'risk' is asserted, not derived: nothing in the event-tree model connects the empirical distribution of CVSS metadata to the probability of a security event. The independence assumption in Eq. 7 is also unjustified for CVSS attributes (e.g., base score and impact score are mechanically related), so P[π] is not even a correct joint frequency. This is a construct-validity failure more fundamental than dataset representativeness: the computed score has no defined target event, so it cannot be validated as a risk estimate without an additional, unprovided link.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes TELSAFE, a hybrid framework for quantitative risk assessment of gaps between security standards and their practical implementation. The framework combines qualitative steps (context definition, risk factor identification, risk analysis) with quantitative steps (risk scenario development via event trees, risk modeling, and risk evaluation). The risk score for a path is Rπ = P[π] × γπ, where P[π] is computed in Algorithm 1 as the product of empirical marginal frequencies of eight CVSS-derived attributes from a Kaggle CVE dataset, and γπ is an impact value derived from CVSS impact scores. A use case for the telecommunications industry is presented, with a stepwise comparison against a purely qualitative ISO 31000 approach for a single CVE.","tokens_in":18762,"tokens_out":4995,"duration_ms":50065,"significance":"The problem addressed is real: organizations need objective, repeatable, and standards-aligned quantitative risk assessment, especially for security gaps arising between standards specifications and implementations. If the framework were sound, it would be a useful contribution to the security risk assessment literature, which the paper reviews. The paper has clear strengths: it surveys academic and industry tools with a structured comparison, it aligns the process with ISO 31000, ISO/IEC 27005, and ITU-T X.1055, it provides a concrete algorithm (Algorithm 1) and a public dataset, and it explicitly acknowledges the lack of public labeled ground-truth data. However, the central quantitative claim is not supported: the computed likelihood P[π] is not a probability of any defined security event, and the risk scores are not validated against any external benchmark. As presented, the framework re-expresses CVSS/EPSS data rather than measuring standards-gap risk.","major_comments":[{"comment":"The likelihood P[π] is computed as the product of empirical marginal frequencies of eight CVSS attributes (base score, exploitability score, EPSS percentile, attack vector, attack complexity, privileges required, user interaction, scope) over the entire Kaggle corpus. This product is not the probability of any event in the risk scenario: the event tree's basic events are vulnerability metadata attributes, not 'a standards-implementation gap occurs' or 'the vulnerability is exploited.' Even if the dataset were a perfect sample of standards-gap vulnerabilities, the product of marginal attribute frequencies would only estimate the probability of observing that attribute combination in the CVE corpus; it would not estimate the likelihood of an incident or a gap-induced compromise. Section IV-A explicitly states the dataset 'has not been acquired by the source as data for gaps between standards and implementation.' Thus the mapping from P[π] to 'risk' is asserted, not derived. This construct-validity failure affects every risk score produced by the framework.","section":"Section IV-C.1, Algorithm 1, Eq. (7)"},{"comment":"The product formula in Eq. (7) is introduced for 'events are independent,' and Algorithm 1 applies it to the eight CVSS attributes without any justification or test of independence. These attributes are not independent in general: CVSS base score is a function of exploitability and impact sub-scores, and EPSS scores are derived from CVE features, so marginal frequencies are mechanically correlated. The product of marginal empirical frequencies therefore does not even give a correct joint frequency for the attribute combination. The paper does not consider the conditional-probability form in Eq. (8), which would require estimating conditional dependencies from data.","section":"Eq. (7) / Algorithm 1 line 20"},{"comment":"The framework is not validated. The paper states that 'the most significant challenge when devising a risk assessment framework is the unavailability of a labeled dataset, i.e., a dataset containing ground truth values for (i) risk assessment, (ii) likelihood, and (iii) impact,' and that 'conventional validation methods relying on labels or ground truth assessments may not be applicable.' However, the risk scores are deterministic transformations of the same CVSS/EPSS data used to define likelihood and impact (Eqs. 9-10 and Algorithm 1), with no external benchmark. The reported example (CVE-2024-7593: likelihood 0.568, impact 0.9782, risk 0.6, level 'Risky') is not compared against any independent measure or validated expert assessment. In the absence of validation, the numerical risk scores are untestable re-expressions of the input attributes rather than predictions of security risk.","section":"Section IV-C.5.c"},{"comment":"The claim that the framework 'eliminates the influence of expert opinion bias' is overstated. Expert judgment is still required in the qualitative steps, including context definition, risk factor identification, and risk analysis; the authors also choose the eight likelihood-influencing attributes, the event-tree structure, and the qualitative risk bands. The statement 'did not require expert opinions to assign any membership functions' is true only in a narrow sense: the framework does not use fuzzy membership functions, but subjective decisions remain in several places. This weakens the paper's positioning relative to the fuzzy-logic-based approaches it criticizes.","section":"Section IV-C.5.a and Section III-A"},{"comment":"The use case does not actually demonstrate application to the telecommunications industry. The Kaggle dataset is a general CVE dataset, and Algorithm 1 is applied to the full corpus without filtering to telecom-related CVEs. The 'context' for CVE-2024-7593 is simply 'Software,' and the quantitative computation uses no telecom-specific assets, threats, or scenarios. The relevance to 5G/6G networks is asserted via the examples in Table III and the alignment with ITU-T X.1055, but it is not operationalized in the risk score computation. As a proof of concept, the use case illustrates the algorithm on generic vulnerability data, not on a telecom-specific risk assessment.","section":"Section IV-A and use case"}],"minor_comments":[{"comment":"The notation for the Cartesian product definition is inconsistent: the text introduces i1y and i2y but later writes elements as i1ai2b with subscripts a, b; clarify the indexing. Also, the phrase 'i1y, y = 1, 2, ..., n1' and 'i2y' appears to have a typo in the second subscript sequence.","section":"Section III-B, Definition 5"},{"comment":"CVE-1999-0199 is shown with Published Date '2020-10-06T13:15Z', which is inconsistent with a CVE published in 1999; verify the data extraction and correct the entry.","section":"Table IV"},{"comment":"The normalization uses γminT, γmaxT, RminT, and RmaxT but these thresholds are never defined. Specify whether they are the observed min/max over the dataset, theoretical bounds, or organization-specific parameters, and state how they are chosen.","section":"Algorithm 1, lines 21 and 23"},{"comment":"The table reports likelihood 0.568 and impact 0.9782 for CVE-2024-7593, but the attribute values for that CVE and the resulting marginal probabilities are not shown. Include the intermediate values so the computation is reproducible.","section":"Section IV-C.4, Table V"},{"comment":"The tool name is spelled inconsistently as 'OCTA VE' and 'OCTAVE' in the text and tables; standardize the spelling throughout.","section":"Section II-B"},{"comment":"Several references are incomplete: [73] has no title, [44] has no author or venue, and [64] and [65] rely on blog-style sources. Complete the bibliography entries and replace non-archival sources where possible.","section":"References"},{"comment":"The phrase 'balanced/unbalanced' dataset is ambiguous; clarify what balancing was performed and how it affects the computed frequencies and risk scores.","section":"Section IV-C.5.b"},{"comment":"Figure 6 is described as a step-wise flow diagram, but the text does not clearly explain the boxes and arrows; ensure the figure is legible and self-contained, and define all abbreviations used in it.","section":"Figure 6"}],"recommendation":"reject","confidential_remarks":"The manuscript's central quantitative claim—that it computes a quantitative risk score for standards-implementation gaps—rests on a likelihood that is not a probability of any defined event. This is a load-bearing construct-validity error that cannot be repaired by editing alone; it would require redefining the target event, obtaining appropriate data, and validating the resulting scores. The absence of any validation and the overstated claim of eliminating expert bias further support rejection. The paper does provide a useful survey and a structured process, but these do not outweigh the fundamental flaw in the risk computation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read the TELSAFE paper. The headline: the framework is a clean, standards-aligned wrapper around a standard risk formula (likelihood × impact), and the use case is honestly labeled as a proof of concept. But the central likelihood term is not the probability of anything in the world, so the quantitative risk scores are not estimates of standards-gap risk.\n\nWhat's new is the specific assembly: ISO 31000/27005/ITU-T X.1055 steps, event-tree language from Papazoglou, and empirical marginal frequencies of CVSS/EPSS attributes drawn from a Kaggle CVE dataset. That combination hasn't appeared in the prior work they cite, and the related work section is careful and fair. I also give them credit for stating plainly that no labeled ground truth exists and that the dataset was not collected to measure standards gaps.\n\nThe soft spot is not the data, it's the math. Algorithm 1 computes P[π] as the product of empirical marginal frequencies of eight CVSS attributes (Eq. 7). That product is the estimated probability of seeing a particular attribute combination in the corpus, nothing more. It is not the probability of an incident, an exploit, or a standards-gap-induced failure. The event-tree definitions they borrow require basic events in a causal chain; here the 'events' are columns of a CVE record. Independence is assumed without justification, which is especially bad because some attributes are mechanically related (CVSS base score is computed from impact and exploitability components). The impact term is similarly derived from the same CVSS data, so the risk score is a deterministic transformation of the input features. There is no target event defined against which the score could be validated, and the paper's own validation discussion concedes that. The single worked example (CVE-2024-7593) can't be reproduced from the text because the dataset and code aren't released.\n\nThat said, these are flaws in a framework proposal, not signs of carelessness. The qualitative steps and the comparison tables are useful. The paper could be repaired: define a concrete target event, use a base rate for gaps from the corpus, and either model dependencies or argue for independence. As is, the quantitative layer overstates what it delivers.\n\nWho gets value? Practitioners who want a structured qualitative risk template aligned to the telecommunication standards. Researchers should read it as a cautionary example of applying event-tree mathematics to non-event features. I'd send it to peer review—a good referee will force the authors to confront the construct-validity issue, and the framework question is worth an airing. My own verdict is reject as a research result until the likelihood term is given a real target event and some external validation.","headline":"The framework is cleanly presented and standards-aligned, but its likelihood term is a product of CVSS attribute frequencies with no defined target event, so the quantitative risk scores are not validated estimates.","tokens_in":19328,"tokens_out":5369,"would_cite":false,"duration_ms":53456,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"TELSAFE computes quantitative risk scores for standards-implementation gaps using event-tree probabilities on CVE data, replacing expert judgment with empirical frequencies.","keywords":["quantitative risk assessment","security gap risk assessment","event tree likelihood modeling","CVE-driven risk assessment","probabilistic modeling","standards-implementation gap","telecommunications security","CVSS"],"falsifier":"Take the subset of CVEs that vendors or advisories explicitly attribute to a standards-implementation gap and run Algorithm 1 on it; if the resulting risk scores do not rank known exploited gap CVEs above the general CVE population, or do not separate known-exploited from non-exploited records, then the framework is measuring generic CVE exposure rather than gap risk.","tokens_in":18291,"feed_emoji":"🛡️","tokens_out":6437,"duration_ms":66665,"temperature":0.7,"pith_summary":"TELSAFE is a hybrid risk assessment framework for the vulnerabilities that arise when security standards are not fully implemented. Its central claim is that the likelihood of such a standards-implementation gap can be quantified from historical vulnerability data using event trees, without expert opinion, and combined with a confidentiality-integrity-availability impact score to yield a numeric risk score $R_\\pi = P[\\pi] \\times \\gamma_\\pi$. If correct, this gives organizations a repeatable, data-driven way to prioritize remediation and to align with ISO 31000, ISO/IEC 27005, and ITU-T X.1055. The demonstration on a public CVE dataset shows the pipeline, though the authors state that the dataset was not acquired specifically as data about standards-implementation gaps.","feed_headline":"Security gaps get numeric risk scores without expert opinions","feed_subtitle":"TELSAFE multiplies data-driven event-tree likelihood by CIA impact to score standards-implementation gaps.","key_machinery":"The central object is the event tree, a diagram whose paths enumerate all combinations of outcomes of $N$ events; the probability of a path is the product, or conditional product, of the outcome probabilities along it. For the CVE use case, eight likelihood-influencing parameters are treated as events, each outcome's probability is its relative frequency in the pre-processed dataset, and the impact $\\gamma_\\pi$ is derived from CVSS impact equations normalized to $[0,1]$. This machinery converts qualitative CVE attributes into a numeric risk score without soliciting expert weights or membership functions.","core_discovery":"The paper claims that a gap between a security standard's specification and its implementation can be modeled as a chain of events, and that the probability of each path can be computed from the empirical frequency of outcome values in a CVE dataset. The risk associated with a path is then the path likelihood times an impact term built from confidentiality, integrity, and availability scores: $R_\\pi = P[\\pi] \\times \\gamma_\\pi$. Because all probabilities come from observed data rather than from expert-chosen membership functions, the authors argue that the framework removes expert-opinion bias and yields identical scores for every user. The framework segments risk assessment into six steps, adding context definition and separate risk-scenario-development and risk-modeling steps to the usual ISO 31000 qualitative flow.","pith_inferences":["The absence of expert opinions covers the probability inputs, but the choice of which CVSS attributes count as likelihood-influencing events is still a modeling decision; testing sensitivity to that choice would show how much of the 'no subjectivity' claim survives.","Because the risk score is a product of likelihood and impact, the method implicitly treats the two as independent; replacing the product with the conditional chain of Eq. 8 would let correlated failures across CVEs be modeled and could change the rankings.","A direct validation route the paper leaves open is to compare TELSAFE scores with the known-exploited flag in public vulnerability data: exploited vulnerabilities should score consistently above non-exploited ones if the likelihood term is meaningful.","The same event-tree construction could be aimed directly at standard specifications, treating each optional security requirement as an event and measuring non-adoption rates from vendor disclosures, which would test the gap hypothesis without relying on a CVE proxy."],"forward_implications":["Adopting TELSAFE lets an organization rank vulnerabilities by a reproducible numeric score, so two analysts given the same dataset will produce the same priorities.","The same six-step pipeline can be applied to any sector with CVE-style vulnerability data, including 5G/6G telecommunications, without retraining models or eliciting expert weights.","Risk scores computed this way can be mapped back onto ISO 31000, ISO/IEC 27005, and ITU-T X.1055 process steps, giving compliance reporting a quantitative artifact.","The framework is extensible to other ISO/IEC 31010 techniques, such as fault trees or Markov analysis, when a single event tree is insufficient to capture a scenario's interactions."],"supporting_citations":[{"why":"Supplies the mathematical foundations of event trees: definitions of event, outcome space, path, and the path-probability product used in Eq. 7.","marker":"[69]"},{"why":"The event-tree quantitative technique the paper adopts for likelihood computation in the quantitative phase.","marker":"[43]"},{"why":"IEC 31010, the standards document that lists event tree analysis among risk assessment techniques and structures the framework's steps.","marker":"[12]"},{"why":"ISO 31000, the generic risk management standard whose assessment phases TELSAFE explicitly aligns with.","marker":"[11]"},{"why":"ISO/IEC 27005, the information-security risk management standard used for the framework's alignment and terminology.","marker":"[2]"},{"why":"ITU-T X.1055, the telecommunications-sector risk management standard that anchors the use case.","marker":"[13]"},{"why":"The data-driven exploit-prediction scoring model whose scores feed the likelihood inputs in the CVE dataset.","marker":"[78]"},{"why":"The vulnerability-database detail record for the CVE used in the worked example comparing qualitative and hybrid outputs.","marker":"[82]"}],"fun_headline_variants":["Data-driven risk scores for security standard gaps","Quantify security gaps with zero expert bias","TELSAFE: event-tree risk scoring from CVE data","Security gaps scored without subjective expert input"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The framework's likelihood numbers inherit whatever the public CVE dataset measures, and that dataset was not collected as a record of standards-implementation gaps, so if it does not represent such gaps the risk scores are not measuring what TELSAFE claims to measure.","fun_headline_variants_meta":{"raw":{"variants":["Data-driven risk scores for security standard gaps","Quantify security gaps with zero expert bias","TELSAFE: event-tree risk scoring from CVE data","Security gaps scored without subjective expert input"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000136,"raw_usage":{"total_tokens":1087,"prompt_tokens":825,"completion_tokens":262,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":441,"completion_tokens_details":{"reasoning_tokens":203}},"tokens_in":441,"tokens_out":262,"duration_ms":3375,"temperature":1.0,"reasoning_tokens":203,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T19:03:22.169879+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the subset of CVEs that vendors or advisories explicitly attribute to a standards-implementation gap and run Algorithm 1 on it; if the resulting risk scores do not rank known exploited gap CVEs above the general CVE population, or do not separate known-exploited from non-exploited records, then the framework is measuring generic CVE exposure rather than gap risk.","supporting_citations":[{"cited_title":"Mathematical foundations of event trees,","cited_arxiv_id":null,"evidence_quote":"Supplies the mathematical foundations of event trees: definitions of event, outcome space, path, and the path-probability product used in Eq. 7."},{"cited_title":"Quantitative risk evaluation based on event tree analysis technique: Application to the design of shield TBM,","cited_arxiv_id":null,"evidence_quote":"The event-tree quantitative technique the paper adopts for likelihood computation in the quantitative phase."},{"cited_title":"Risk management — Risk assessment techniques,","cited_arxiv_id":null,"evidence_quote":"IEC 31010, the standards document that lists event tree analysis among risk assessment techniques and structures the framework's steps."},{"cited_title":"Risk management — Guidelines,","cited_arxiv_id":null,"evidence_quote":"ISO 31000, the generic risk management standard whose assessment phases TELSAFE explicitly aligns with."},{"cited_title":"Risk management and risk profile guidelines for telecommunication organizations,","cited_arxiv_id":null,"evidence_quote":"ITU-T X.1055, the telecommunications-sector risk management standard that anchors the use case."},{"cited_title":"[Online]","cited_arxiv_id":null,"evidence_quote":"The data-driven exploit-prediction scoring model whose scores feed the likelihood inputs in the CVE dataset."},{"cited_title":"[Online]","cited_arxiv_id":null,"evidence_quote":"The vulnerability-database detail record for the CVE used in the worked example comparing qualitative and hybrid outputs."}],"review_version":1}