{"id":"6103402c-8b7c-4d6f-90ee-dfb477d8579b","arxiv_id":"2508.13364","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"Adding scraped OSINT data and an AlienVault pulse-count term to HAL 9000's risk score is claimed to produce more secure configuration recommendations, but the reported experiments do not directly test the scraped-data contribution.","lead":"This paper adds an automated web scraper and new threat-intelligence sources to HAL 9000, a machine-learning risk manager for intrusion-tolerant systems. It also adjusts the risk formula to count how often a vulnerability appears in AlienVault threat pulses, then simulates whether the upgraded system recommends safer configurations.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper never measures the contribution of the new scraper sources: §4.1 restricts the comparison dataset to NVD/ExploitDB, so Eq. 7's pulse term is identically zero in the only Risk Manager evaluation; the claimed improvement rests on one arithmetic example.","rationale":"The engineering contribution is real and partly reproducible: the scraper is publicly available, written in Rust, and Table 4 reports concrete resource measurements. My concern is specifically the research claim that integrating scraper-derived intelligence substantially improves HAL 9000's risk management. The paper's own experimental setup limits the comparison dataset to NVD and ExploitDB, and the comparison graphs are scored with the Lazarus method rather than with Eq. 7, so the new pulse-based scoring term is never exercised in the evaluation. The only numeric demonstration of Eq. 7 is a single illustrative CVE, with no calibration against observed exploitation. Because the Configurator optimizes Eq. 8 and Eq. 9 using hal_score, the recommended-configuration results cannot be attributed to the new data sources without an ablation that includes pulse counts and OSV data. This does not change the reader's REJECT verdict, but it identifies the specific experiment needed to convert the engineering description into a supported claim.","tokens_in":18514,"tokens_out":7346,"duration_ms":74925,"concrete_test":"Re-run the Risk Manager comparison using the scraper's actual local database without the §4.1 NVD/ExploitDB restriction: populate the dataset with OTX pulse counts and OSV entries for the same 16 OSs, let HAL select configurations with Eq. 7, and compare its security and resilience scores against a HAL variant using Eq. 6 (no pulse term), scoring both under Eq. 7 and under the Lazarus baseline. If adding the pulse term changes no recommended configuration, or if hal_score does not improve discrimination of CVEs listed in CISA's Known Exploited Vulnerabilities on held-out data, the paper's claim that scraper-derived intelligence substantially improves risk management fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that integrating scraper-derived intelligence (AlienVault OTX, OSV) substantially improves HAL 9000's risk management. The only mechanism for that improvement is the log(#related_pulses) term in Eq. 7, yet the paper's main risk-manager experiment deliberately excludes the new sources: §4.1 states that the dataset was limited to NVD and ExploitDB to ensure a fair comparison with the other Risk Managers. With no AlienVault pulse counts in the data, hal_score(v) reduces to the earlier Eq. 6, so Figures 5 and 6 cannot show any effect of the extension. Moreover, those figures score all configurations using Lazarus's method rather than Eq. 7, because the paper says the Lazarus calculation was applied for comparison purposes. The only quantitative support for Eq. 7 is the CVE-2017-11882 example, where the score rises from 7.2 to 8.9 by arithmetic. No evidence is provided that #related_pulses correlates with exploitation likelihood, and the additive log term has no derivation or normalization against the weighted CVSS product: for one pulse it contributes zero, while for large pulse counts it can dominate the capped 0-10 score. Consequently, the reported more secure configurations may reflect HAL's earlier scoring and clustering choices rather than scraper-derived intelligence, leaving the central claim untested by the reported experiments.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper extends HAL 9000, an intrusion-tolerant-system risk manager previously introduced by the same group, with two additions: a custom scraper that periodically harvests vulnerability and exploit data from NVD, ExploitDB, AlienVault OTX, and OSV into a local PostgreSQL database, and a revised hazard scoring formula, Eq. (7), which appends the term log(#related_pulses) to the EPSS-weighted score so that a CVE's presence in AlienVault threat pulses raises its contribution to configuration risk. The architecture consists of four components (CVSS score prediction, description-based clustering, score reassessment, and a Configurator that recommends OS configurations minimizing security_risk and resilience_risk as defined in Eqs. (8)-(9)). The experiments compare clustering/preprocessing choices by their effect on HAL's own risk calculation (Figure 4), compare HAL against the Lazarus and Heo risk managers on a simulated 2023 deployment whose dataset is explicitly limited to NVD and ExploitDB entries (Figures 5-6), compare three CVSS prediction models (Table 3), and measure scraper retrieval time and entry counts (Table 4). The abstract claims that integrating scraper-derived intelligence substantially improves HAL 9000's ability to address emerging threats.","tokens_in":18768,"tokens_out":12451,"duration_ms":113252,"significance":"If the headline claim were established, the paper would offer a useful, deployable pattern for ITS risk management: automated ingestion of multiple OSINT sources with a risk adjustment driven by real-world exploitation chatter. There are genuine strengths to credit: the scraper is a concrete artifact (1,268 lines of Rust, publicly hosted), the paper candidly documents API rate limits and scraping obstacles (notably ExploitDB's anti-scraping protections and OTX's hourly limits), and the CVSS prediction comparison in Table 3 is a fair replication of three published methods with reported RMSE and runtime. However, the specific extension — the pulse-count term in Eq. (7) and the new data sources — is exactly the portion that the evaluation does not exercise: Section 4.1 restricts the comparative dataset to NVD/ExploitDB and Section 4.2 scores all configurations with the Lazarus formula, so Figures 5 and 6 cannot show any effect of the paper's contribution, and the only quantitative support for Eq. (7) is a single arithmetic example. As an engineering description of an OSINT scraper the paper is informative; as a validation of a new risk-scoring mechanism it is not.","major_comments":[{"comment":"The paper's central claim — that integrating scraper-derived intelligence 'substantially improves' HAL's risk management — is not tested by the reported experiments. Section 4.1 explicitly limits the comparative dataset to NVD and ExploitDB 'to ensure a fair comparison', so AlienVault pulse counts and OSV records are absent, and the log(#related_pulses) term in Eq. (7) is identically zero in the only Risk Manager evaluation. Section 4.2 further states that 'both graphs use the Lazarus calculation method for the scores presented', so even HAL's own Eq. (7) score is not the quantity plotted. The improvements visible in Figures 5 and 6 therefore reflect the clustering and configuration-selection machinery inherited from prior work, not the new data sources or the new scoring term, and the abstract's claim rests solely on the CVE-2017-11882 example, where the score rises from 7.2 to 8.9 by arithmetic. An evaluation in which the new sources are actually present, and in which the contribution of the pulse term is isolated (e.g., with vs. without Eq. 7), is required to support the central claim.","section":"§4.1, §4.2, Figs. 5–6"},{"comment":"The new scoring term log(#related_pulses) is introduced without derivation, calibration, or validation. The paper gives no evidence that a CVE's count of AlienVault pulses correlates with exploitation likelihood, and no normalization anchors the dimensionless pulse count against the weighted CVSS product in Eq. (6); the logarithm base is unspecified (the CVE-2017-11882 example, 7.2 + log(50) = 8.9, implies base 10), and the behavior at #related_pulses = 0 is undefined since log(0) is undefined — a case that necessarily arises in the Section 4.1 dataset, which contains no pulse data at all. With the cap of 10, the term can saturate the score entirely: for the example's pre-pulse value of 7.2, a base-10 log reaches the cap at roughly 630 pulses, allowing chatter volume to dominate the hazard estimate. The single illustrative example cannot validate the formula.","section":"Eq. (7), §3"},{"comment":"The selection of OPTICS with sentence embeddings is circular with respect to the paper's performance claim. Figure 4 evaluates clustering algorithms by their 'subsequent effects on the HAL risk calculation (lower is better)', and the text concludes that OPTICS and DBSCAN are best because they 'provide the lowest risk scores' under HAL's own formula — the same formula that the Configurator in Eqs. (8)–(9) optimizes. An algorithm that depresses the system's objective function is therefore guaranteed to make the system look good in the downstream comparison of Figures 5–6, where configurations are chosen by that very objective. No external clustering-quality measure (e.g., entropy, homogeneity, or adjusted Rand index against a ground truth) and no independent validation of the recommended configurations is reported, so the conclusion in Section 5 that HAL's clustering 'yields better groupings than prior approaches' is unsupported.","section":"Figure 4, §4.2"},{"comment":"The generated dataset raises reliability concerns. Table 1 lists 'Ubuntu 16.04 / 2,035' twice, and identical CVE counts appear across many OS versions (Debian 6, 7, 8 and 10 all at 3,923; Ubuntu 10.04 through 22.04 all at 2,035; Fedora 16, 24, 30 and 38 all at 835; Solaris 10 and 11 both at 359), which suggests the generator assigns counts per OS family rather than per version; because Figures 5–6 are computed entirely from this dataset, per-version granularity matters for the validity of the comparison. Additionally, Table 4 reports exactly the same #Entries value (277,152) for NVD and AlienVault OTX, which is implausible and appears to be a copy error. The authors should document the generator's assignment logic and correct Table 4.","section":"Tables 1–2, Table 4"}],"minor_comments":[{"comment":"The cross-references to equations are inconsistent: Section 3 says the hal_score(v) is 'presented in equation 4' and 'as shown in equation6', but the displayed hal_score equations are numbered (6) and (7), while Eq. (4) is the second patched(v) formula from Section 2.1; Section 3.1 step 3 refers to 'equations 1 to 6', which does not match the displayed numbering (5)–(7). These references should be corrected.","section":"§3, Eqs. (4)–(7)"},{"comment":"Equation (4) is labeled patched(v) = 1.25^v.exploited, but the right-hand side depends on v.exploited and not on v.patched; it should be the exploited(v) factor consistent with the product in Eq. (1). The label appears to be a typo that makes the scoring system hard to follow.","section":"§2.1, Eq. (4)"},{"comment":"The prose describes the resilience score as 'the multiplication between the reassessed CVSS score of the common CVEs and respective EPSS score', but Eq. (9) sums hal_score(v), which already embeds the EPSS weighting inside hal_score; the descriptive text and the equation should be reconciled.","section":"§4.2, resilience score"},{"comment":"The conclusion states that HAL is 'demonstrably superior to traditional systems in terms of automation, accuracy, and adaptability' and that experimental results 'demonstrate that HAL's clustering of CVEs yields better groupings than prior approaches'; these statements outrun the evidence in Section 4, where no accuracy metric for clustering against a ground truth and no test of the pulse term are reported.","section":"Section 5, Conclusion"}],"recommendation":"reject","confidential_remarks":"The paper is an extension of the authors' own previously accepted HAL 9000 paper, and the genuinely incremental pieces — the scraper and Eq. (7) — are exactly the parts that are unevaluated. The most serious problem is evaluative circularity: the system's quality is measured by the objective function its own Configurator optimizes, and the clustering algorithm is selected for minimizing that same function. If the authors can add an evaluation that includes the new data sources, isolates the effect of the pulse term, and validates pulse counts against an independent exploitation signal (e.g., EPSS or observed incident data), the work could become publishable; as submitted, the headline claim is unsupported. The identical #Entries values for NVD and AlienVault OTX in Table 4 and the duplicated OS rows in Tables 1–2 may indicate data-handling errors that should be checked before any resubmission."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe new thing here is real: a multi-source OSINT scraper feeding HAL 9000, plus a pulse-count term in the risk formula. The scraper design is described in detail, and the code is on GitHub. The clustering comparison and the CVSS prediction experiments are competently done. Researchers building ITS risk managers will find the architecture description useful.\n\nThe problem is that the paper's central claim—that scraper-derived intelligence substantially improves risk management—is never tested. Section 4.1 explicitly limits the comparative dataset to NVD and ExploitDB to ensure fair comparison with the other Risk Managers. That means the new sources (AlienVault OTX, OSV) do not appear in the only experiment that evaluates risk-manager quality. On top of that, Figures 5 and 6 score every configuration using Lazarus's method, not the new Eq. 7, so even HAL's results are not showing the extension. The only quantitative support for Eq. 7 is the CVE-2017-11882 example, where the score rises from 7.2 to 8.9 by arithmetic. There is no evidence that pulse counts correlate with exploitation likelihood, and the additive log term sits awkwardly on top of a weighted CVSS product.\n\nThere is also a circularity in the clustering choice: OPTICS and DBSCAN are selected because they give the lowest risk scores under HAL's own formula, and the Configurator then optimizes that same score. That makes the \"more secure configurations\" claim weaker than it looks.\n\nI want to be fair: the paper is an honest engineering description. It does not hide the dataset limitation; it just overclaims in the abstract and conclusion. If the authors narrowed the claim to \"we built a scraper and integrated it, with no end-to-end validation yet,\" this would be a solid systems paper. As it stands, the central causal claim is unsupported.\n\nI'd send it to a serious referee—there is enough substance in the system and the clustering work to merit a careful look—but the referee report would have to demand either a direct evaluation of the scraper's contribution or a substantial narrowing of the claims. I wouldn't cite it in my own work until that happens.","headline":"The scraper and pulse-count term are real additions, but the paper's only risk-manager comparison excludes the new sources, so the central claim rests on a single arithmetic example.","tokens_in":19366,"tokens_out":2707,"would_cite":false,"duration_ms":25776,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that adding continuously scraped OSINT feeds and a pulse-aware logarithmic term to HAL 9000's risk score lets an intrusion-tolerant system recommend configurations with lower computed risk than two published baseline…","keywords":["intrusion tolerant systems","risk management","CVSS score prediction","OSINT scraper","AlienVault OTX","EPSS","CVE clustering","adaptive configuration"],"falsifier":"Take the paper's simulated 2023 dataset, recompute HAL's recommended configuration after setting the number of related pulses to zero for every CVE, and compare the resulting security and resilience scores against the two baselines; if the advantage disappears or reverses, the improvement comes entirely from the pulse term. Alternatively, correlate each CVE's pulse count with actual exploitation within 30 days, for example through EPSS or incident reports, across a large sample; a near-zero or negative correlation would undercut the assumption that the log term measures real-world risk.","tokens_in":18275,"feed_emoji":"🛡️","tokens_out":8791,"duration_ms":88640,"temperature":0.7,"pith_summary":"The paper claims that an intrusion-tolerant system's risk manager becomes substantially more responsive to new threats when it continuously scrapes additional public threat feeds instead of relying on NVD and ExploitDB alone. The extended HAL 9000 automatically ingests AlienVault OTX and OSV data, predicts CVSS scores for unassessed CVEs, and reassesses each vulnerability with a new score that adds $\\log(\\#related\\_pulses)$ when a CVE appears in AlienVault pulses. In a simulated 2023 deployment across sixteen operating systems, the configurations HAL recommends have lower computed security risk and resilience risk than the two earlier risk managers on several months, while occasionally trading away some resilience for security. If correct, the contribution is a concrete route to narrowing the window between vulnerability disclosure and automated defensive action.","feed_headline":"AlienVault pulse counts sharpen intrusion-tolerance risk scoring","feed_subtitle":"Continuous scraping of four threat feeds plus a pulse-aware score beats two published baselines on simulated 2023 data.","key_machinery":"The central object is the revised risk score in Eq. (7), a hazard formula that combines a Lazarus-style multiplicative CVSS score with EPSS probability weighting and a new logarithmic \"threat attention\" term. The work it does is to make the Configurator's objective function, the sums of these scores over nodes and over shared or clustered CVEs, sensitive to active discussion of a CVE in curated AlienVault pulses. Supporting machinery includes a Rust scraper that writes NVD, ExploitDB-via-searchsploit, AlienVault OTX, and OSV data into a local PostgreSQL store; a random-forest CVSS predictor for unscored CVEs; OPTICS clustering, a density-based clustering algorithm robust to outliers, applied to sentence embeddings to find descriptions that describe the same vulnerability; and the Configurator that enumerates operating-system combinations and recommends the one minimizing security and resilience risk, prioritizing resilience.","core_discovery":"The paper's central claim is that \"integrating scraper-derived intelligence with HAL 9000's risk management framework substantially improves its ability to address emerging threats,\" and its evidence is a head-to-head comparison of recommended configurations. In the revised scoring equation, $\\mathit{hal\\_score}(v) = \\min(10, \\mathit{score}(v)(1-\\mathit{EPSS}(v)) + \\mathit{score}(v)_{wp}\\mathit{EPSS}(v) + \\log(\\#related\\_pulses))$, the new last term is the load-bearing change: it raises a vulnerability's accepted risk according to how many AlienVault OTX pulses mention it. The paper illustrates with CVE-2017-11882, whose score is 3.65 under the earlier calculation, 7.2 in HAL's prior weighted formulation, and 8.9 once the 50 related pulses are counted, placing it near \"Critical\" despite having a patch. The experiments then show that across 2023 the configurations selected by the extended HAL carry lower security and resilience scores than the baselines in the months shown, supporting the claim that the added data source improves risk management.","pith_inferences":["The logarithmic pulse term is an untested functional form; a natural follow-up is to estimate its coefficient or replace it with a calibrated attention weight from historical exploitation data rather than fixing the increment as $\\log(\\#related\\_pulses)$.","The same scraper architecture could be extended to dark-web forums, vendor advisories, and social media with a weighted \"threat attention\" score, but then misinformation and fake-consensus manipulation, which the paper acknowledges as future work, would become a first-order concern.","The paper's \"more secure configurations\" claim is about computed risk scores, not realized compromises; a testable implication is that deployments using HAL should show fewer successful intrusions over time than deployments using the baselines."],"forward_implications":["If HAL's claim holds, an ITS can ingest four public feeds automatically, so newly disclosed CVEs receive a predicted score and a reconfigured risk evaluation within hours rather than after NVD's manual analysis, which the paper notes can lag by a year.","A CVE with a high AlienVault pulse count can be pushed into a higher severity band even when a patch exists, giving the Configurator a reason to avoid configurations that include that software.","Because the Configurator optimizes the pulse-aware score, the recommended configurations can differ from those a pure CVSS-sum minimizer would choose; in the paper's April example, HAL accepts higher resilience risk to obtain lower security impact.","The scraper's measured runtimes and API limits suggest that periodic hourly updates are feasible, with AlienVault OTX as the bottleneck, making continuous risk reassessment practical for a deployed ITS."],"supporting_citations":[{"why":"Supplies the Lazarus risk manager and its multiplicative scoring equation that HAL extends, and the baseline for resilience comparisons.","marker":"[8]"},{"why":"Supplies the diversity-policy risk manager by Heo et al. used as a comparison baseline for security score.","marker":"[13]"},{"why":"Supplies the method for predicting CVSS scores from vulnerability descriptions, adopted as HAL's score predictor after experiments.","marker":"[25]"},{"why":"Supplies the EPSS probability of exploitation within 30 days, central to both weighted equations in HAL's scoring.","marker":"[59]"},{"why":"Supplies AlienVault OTX pulses, the source of the new logarithmic term in the revised score.","marker":"[64]"},{"why":"Supplies the NVD database with CVSS base scores and CVE records that ground the entire risk calculation.","marker":"[62]"},{"why":"Supplies searchsploit as the integration path for ExploitDB data, a source used by HAL and by the compared managers.","marker":"[61]"},{"why":"Supplies the OSV database, one of the newly scraped sources added by this extension.","marker":"[65]"}],"fun_headline_variants":["HAL 9000 adds AlienVault pulse counts to sharpen intrusion risk scoring","Pulse-aware scoring boosts HAL 9000's threat detection","AlienVault pulses raise unpatched CVE risk scores in HAL 9000","HAL 9000 mines threat feeds to outscore baselines on 2023 data"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the number of AlienVault pulses mentioning a CVE is a reliable proxy for real-world exploitation risk; the $\\log(\\#related\\_pulses)$ term is asserted with only a single illustrative example, and if pulse counts do not correlate with exploit likelihood, the claimed security improvement in the recommended configurations collapses.","fun_headline_variants_meta":{"raw":{"variants":["HAL 9000 adds AlienVault pulse counts to sharpen intrusion risk scoring","Pulse-aware scoring boosts HAL 9000's threat detection","AlienVault pulses raise unpatched CVE risk scores in HAL 9000","HAL 9000 mines threat feeds to outscore baselines on 2023 data"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000329,"raw_usage":{"total_tokens":1908,"prompt_tokens":1087,"completion_tokens":821,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":703,"completion_tokens_details":{"reasoning_tokens":734}},"tokens_in":703,"tokens_out":821,"duration_ms":7550,"temperature":1.0,"reasoning_tokens":734,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T17:13:52.594451+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the paper's simulated 2023 dataset, recompute HAL's recommended configuration after setting the number of related pulses to zero for every CVE, and compare the resulting security and resilience scores against the two baselines; if the advantage disappears or reverses, the improvement comes entirely from the pulse term. Alternatively, correlate each CVE's pulse count with actual exploitation within 30 days, for example through EPSS or incident reports, across a large sample; a near-zero or negative correlation would undercut the assumption that the log term measures real-world risk.","supporting_citations":[{"cited_title":"Lazarus: Automatic management of diversity in bft systems","cited_arxiv_id":null,"evidence_quote":"Supplies the Lazarus risk manager and its multiplicative scoring equation that HAL extends, and the baseline for resilience comparisons."},{"cited_title":"Designing and implementing a diversity policy for intrusion-tolerant systems","cited_arxiv_id":null,"evidence_quote":"Supplies the diversity-policy risk manager by Heo et al. used as a comparison baseline for security score."},{"cited_title":"An automatic method for CVSS score prediction using vulnerabilities description.Journal of Intelligent & Fuzzy Systems","cited_arxiv_id":null,"evidence_quote":"Supplies the method for predicting CVSS scores from vulnerability descriptions, adopted as HAL's score predictor after experiments."},{"cited_title":"Exploit prediction scoring system (epss)","cited_arxiv_id":null,"evidence_quote":"Supplies the EPSS probability of exploitation within 30 days, central to both weighted equations in HAL's scoring."},{"cited_title":"https://otx.alienvault.com","cited_arxiv_id":null,"evidence_quote":"Supplies AlienVault OTX pulses, the source of the new logarithmic term in the revised score."},{"cited_title":"The national vulnerability database (nvd): Overview","cited_arxiv_id":null,"evidence_quote":"Supplies the NVD database with CVSS base scores and CVE records that ground the entire risk calculation."},{"cited_title":"https://www.exploit-db.com/searchsploit","cited_arxiv_id":null,"evidence_quote":"Supplies searchsploit as the integration path for ExploitDB data, a source used by HAL and by the compared managers."},{"cited_title":"OSV - Open Source Vulnerabilities.; 2025","cited_arxiv_id":null,"evidence_quote":"Supplies the OSV database, one of the newly scraped sources added by this extension."}],"review_version":2}