{"id":"f5754809-2c2d-4d24-ba9a-a92388715313","arxiv_id":"2506.06604","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Website technology fingerprints crawled from a domain predict whether that organization appears in public data breach incident reports, achieving ROC AUCs between 87% and 92% on within-dataset tests.","lead":"This paper trains a machine learning model to estimate an organization's risk of a data breach using only technology signatures collected by crawling its website, instead of requiring Internet scans of its IP addresses. The approach matters because it extends cyber risk scoring to millions of small and medium-sized organizations that lack identifiable network assets.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported AUCs may separate breach-reporting organizations from the privacy-policy-filtered Tranco population by size and sector, not by cybersecurity posture; the paper's own defense of within-peer-group separation is asserted but never demonstrated.","rationale":"The reader identified the same weakest assumption, and I agree. The paper's internal evidence is solid as far as it goes: within-dataset AUCs are high (Fig. 3), BFSR 24 held-out performance remains around 86-88% (Fig. 7), calibration is good (Fig. 6), and the Wayback experiment (Fig. 11) is an honest attempt at temporal simulation. None of these, however, addresses the composition of the negative pool relative to the positives. A held-out set drawn from the same biased positive source (BFSR 24) cannot validate external validity for the much broader and smaller organization population the paper motivates. The authors explicitly acknowledge the different-pools problem and cite Woods and Bohme, but their stated mitigation is an assertion rather than a demonstrated result: Section 6.2's claim of 'substantial risk separation among firms of the same type' is not backed by any within-stratum analysis. The concrete test proposed above would settle this. If it fails, the appropriate disposition would shift toward rejection or at minimum a stronger condition requiring a matched-size experiment; if it passes, conditional acceptance can be upgraded. Since the paper itself frames the result as correlational and acknowledges the limitation, the existing CONDITIONAL verdict is the right default, so I recommend UNCHANGED.","tokens_in":19260,"tokens_out":5106,"duration_ms":55916,"concrete_test":"Build a size-matched evaluation set. For each positive domain, obtain an employee-count or revenue estimate from a firmographics source (or, failing that, from manual review of About Us/company pages). From Tranco domains that pass the paper's privacy-policy filter, sample one or more negatives in the same size band (e.g., decile or log-band of employee count) and, ideally, same NAICS sector. Re-run the authors' exact feature-extraction and 5-fold XGBoost pipeline on positives plus this matched negative set, and report ROC AUC. If the matched AUC falls substantially below the reported 86-92% (e.g., below about 0.65), the original separation is largely attributable to size/sector confounds rather than cyber risk; if matched AUC remains high, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing assumption is that the negative pool is a valid comparison population for the breach-positive pool. Positives are drawn from VCDB and BFSR, which the paper itself (Section 6.2) concedes are biased toward larger organizations and disclosure-prone sectors. Negatives are 3,971 Tranco domains that survived two filters: a live crawl must succeed and a privacy page must exist (Section 3.3.2). Both filters are correlated with organizational formalization and size: small firms and personal sites are differentially removed, leaving larger, legally exposed entities. Because the technology-signature features (CMS, analytics, CDN, cookie-compliance tools, etc.) are also strongly correlated with website sophistication and budget, XGBoost can achieve high AUC by separating 'big/sophisticated' from 'small/simple' rather than 'breached' from 'unbreached.' Section 6.2 asserts that 'there is substantial risk separation among firms of the same type,' but no size-stratified, matched-control, or within-peer-group result is provided anywhere in the paper, so this key defense is unsubstantiated. The headline 86.7-92.3% AUCs and the claim that the model can score millions of SMEs therefore rest on an untested population-matching assumption. If size or sector explains most of the separation, the instrument is not measuring what the title promises for the SME population. Relatedly, the >95% yield claim is computed only on positive/incident domains with known websites, not on the target Tranco-like population, where the end-to-end yield is 3,971/10,000 = 39.7%.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a cyber risk assessment approach that replaces Internet scan features with technology signatures collected by crawling an organization's website. Positive labels are drawn from VCDB (2022-23) and BlackFog ransomware reports (BFSR 22-23, with BFSR 24 held out), and negatives are sampled from Tranco domains that pass a live-crawl and privacy-policy filter. Features are one-hot-encoded technology names/versions plus category counts, pruned to 1,013 features, and modeled with XGBoost under 5-fold cross-validation. The authors report within-dataset ROC AUCs of 92.3% (VCDB), 86.7% (BFSR 22-23), and 89.2% (combined), a held-out BFSR 24 AUC of 86.7%, well-behaved calibration, SHAP-based feature importances, and performance gains from adding NAICS sector information. They also report a >95% data-collection yield and a Wayback Machine experiment simulating a predictive setting.","tokens_in":19440,"tokens_out":4394,"duration_ms":48553,"significance":"If the central claim holds, the paper makes a useful contribution: it demonstrates a new, low-cost data modality for cyber risk scoring that avoids the IP-attribution bottleneck of scan-based methods and could extend coverage to organizations without dedicated IP assets. The authors should be credited for several strengths: independent label sources, a held-out BFSR 24 set, manual validation of the automated domain-mapping step, calibration plots, SHAP analyses, and a Wayback Machine control for temporal alignment. The main risk is that the reported discrimination may reflect organizational size and sector rather than cybersecurity posture, because the positive and negative pools differ systematically. That issue is load-bearing for the paper's central claim and for the promise of scoring SMEs, and it is only addressed by an unsubstantiated assertion in Section 6.2.","major_comments":[{"comment":"The negative pool is constructed from Tranco domains that survive a live-crawl filter and a privacy-policy filter (Section 3.3.2), while positives come from VCDB and BFSR incident reports that, as the paper concedes in Section 6.2, are biased toward larger organizations and disclosure-prone sectors. Because technology-signature features are also correlated with website sophistication and budget, the high AUCs may largely separate 'large/sophisticated' from 'small/simple' organizations rather than breached from non-breached ones. Section 6.2 asserts that 'there is substantial risk separation among firms of the same type,' but no size-stratified, sector-stratified, or matched-control result is provided anywhere in the paper. Please add within-sector or size-stratified AUC analyses, or a matched-control experiment (e.g., matching positives to negatives on NAICS sector and a firm-size proxy), and report how much discrimination remains.","section":"Section 3.3 and Section 6.2"},{"comment":"The >95% yield claim is computed over positive samples whose domains were already resolved and known to host websites (817/830 for VCDB and 1,322/1,366 for BFSR 22-24). For the Tranco negative population, only 3,971 of 10,000 domains (39.7%) pass both the crawl and privacy-policy filters. The paper's stated contribution (2) that this 'model can be applied to the millions of small entities' is therefore not supported by the reported yield, since coverage among SMEs without a detectable privacy policy or with a non-crawlable website may be much lower. Please report coverage on a population-representative sample stratified by organization size, or revise the yield and applicability claims to reflect the actual denominator.","section":"Section 3.4 and Contribution (2)"},{"comment":"The BFSR 24 evaluation is presented as evidence of robustness over time, but the features for BFSR 24 domains are collected by live crawls at the time of the study, after the 2024 incidents occurred. The paper itself states in Section 6.3 that the crawled data is not temporally aligned with incident dates. Thus, the BFSR 24 AUC demonstrates generalization to a new label source with contemporaneous features, not predictive performance in a true time-lapse setting. Please either restrict the temporal-robustness claim to the Wayback Machine experiment in Section 6.3, or clearly label the BFSR 24 result as a contemporaneous-feature transfer evaluation.","section":"Section 4.2.3 and Section 6.3"},{"comment":"The calibration plot is computed on the constructed positive/negative pools, whose ratio and composition are determined by the sampling design rather than by any population base rate. Because the negative pool is a privacy-policy-filtered Tranco sample, the empirical positive rate in each bin is an artifact of the sampling scheme. The statement that the classifier outputs are 'well-calibrated' and require no further calibration can therefore support only within-sample calibration, not calibrated breach probabilities for arbitrary organizations. Please add a prevalence-adjusted discussion or soften the absolute probability interpretation.","section":"Section 4.2.3, Figure 6"}],"minor_comments":[{"comment":"There is a typo in the opening sentence: 'we leveragesupervised learning' should read 'we leverage supervised learning.'","section":"Section 2"},{"comment":"'Table 2 summaries the curated datasets' should be 'Table 2 summarizes the curated datasets.'","section":"Section 3.3, Table 2 caption"},{"comment":"The description of SHAP contributions is confusing: the text says HTTP/3 and Cloudflare are 'positive contributors to lower estimated risk' and Siteimprove and Simpli.fi are 'negative contributors,' which inverts the usual sign convention. Please state the sign convention explicitly (e.g., positive SHAP pushes output toward the positive class).","section":"Section 4.2.2"},{"comment":"The legends contain apparent typos such as 'B th → B th' and oddly spaced labels like 'F alse p sitive rate' and 'T rue p sitive rate'; please correct these in the final figures.","section":"Figure 3 and Figure 7 legends"},{"comment":"The paper cites Woods and Böhme [46] for the concern that the model may distinguish large corporations from smaller organizations, but it does not discuss what fraction of the separation is attributable to size/sector even after including sector as a feature in Section 5. Reporting the sector-only AUC together with the combined AUC is useful, but an explicit decomposition of variance or an interaction analysis would clarify the residual role of technology features.","section":"Section 6.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for cs.CR and the underlying idea is promising, but the referees' main concern about population matching is real and is acknowledged internally in Section 6.2. The revision should focus on adding the missing stratified or matched analyses rather than on prose fixes. I would not reject, because the central approach is defensible and the required evidence is obtainable from the same data."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nRead through 'Scoring the Unscorables.' The headline numbers are real: domain-level technology signatures, collected from a website crawl, do separate the breach/ransomware positives from the Tranco negatives with AUCs in the high 80s to low 90s, and the held-out 2024 set at 86.7% AUC is a genuine out-of-time check. That is a useful contribution. The paper is careful about evaluation: 5-fold CV, calibration plots, SHAP, and a Wayback-based historical check. It also is unusually honest—Section 6.2 names the positive/negative pool mismatch and cites Woods and Böhme on exactly that failure mode. I value that.\n\nThe soft spot is the one the reader flagged and it is real: the negative pool is 3,971 Tranco domains that allow a crawl and have a privacy page. Small and informal sites get filtered out, and the positives come from VCDB/BFSR, which skew toward larger and disclosure-prone organizations. The technology features (CMS, analytics, CDN, cookie-compliance) correlate with website sophistication and budget. So the model may be separating 'big and formal' from 'small and simple' more than 'breached' from 'unbreached.' The paper says there is 'substantial risk separation among firms of the same type' but gives no stratified or matched-control result to back it. That is the weakest paragraph in the paper.\n\nThe yield claim also needs a flag. The >95% yield is on incident-report domains. On the random Tranco sample, the end-to-end yield is 3,971/10,000, about 40%. The abstract and contribution list imply the method covers millions of SMEs at high yield; the data do not support that yet.\n\nNone of this kills the core result: if you want a ranking of organizations by breach-likelihood within a similar size/sector population, the crawl features work. What is not established is the absolute probability calibration or the applicability to true SMEs. The paper is a decent step, not a finished instrument.\n\nWho should read it? People working on cyber risk scoring, insurance underwriting, and data-driven security. It deserves a serious referee—conditional acceptance with a request for a matched-control or size-stratified evaluation. I would not desk-reject it.","headline":"Honest, well-evaluated paper with a real new data source, but the headline AUCs may partly reflect a size/sector confound that the paper asserts but never demonstrates.","tokens_in":20113,"tokens_out":2698,"would_cite":true,"duration_ms":28706,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a cyber risk model built from crawlable website technology signatures—not Internet scans—can estimate an organization's likelihood of a data breach, with within-corpus ROC AUC up to 92.3% and 86.7% on a held-out…","keywords":["cyber risk quantification","data-driven security","cyber incidents","ransomware","machine learning","web crawling","data breach prediction"],"falsifier":"Construct a matching test set in which each breached organization is paired with a non-breached organization of the same sector and comparable size, and recompute the model's AUC: if discrimination falls toward chance on these matched pairs, the score is capturing demographic sorting rather than the security posture claimed for the technology signatures.","tokens_in":18923,"feed_emoji":"🛡️","tokens_out":8548,"duration_ms":85154,"temperature":0.7,"pith_summary":"The paper sets out to show that an organization's cyber risk can be scored from the technologies detectable on its public website, without needing to scan its IP addresses or attribute network assets to the organization. The authors crawl each domain, record digital signatures of the software and services it uses, and train a supervised model against reports of data breaches and ransomware incidents. They report that the resulting scores separate breached from non-breached organizations with area-under-the-ROC-curve values of 92.3 percent (general incidents), 86.7 percent (ransomware incidents), and 89.2 percent combined, and that the model holds at 86.7 percent on a 2024 set of ransomware incidents completely withheld from training. If this stands, cyber risk scoring becomes applicable to the millions of small and medium organizations for whom IP-based scans are unavailable or incomplete, because the method only requires a website that can be crawled.","feed_headline":"Website technology signatures score breach risk at 92 percent","feed_subtitle":"Crawlable web data delivers risk scores to millions of small firms that IP scans cannot reach.","key_machinery":"The paper's machinery is a crawl-based technology-signature pipeline. A crawler visits the organization's homepage, up to nine random internal pages, and up to nine pages containing 'privacy' in the link; from these pages it records the presence and versions of web technologies (content management systems, JavaScript libraries, CDNs, analytics trackers, payment processors, security and cookie-compliance tools, and similar). Each technology name and version becomes a binary feature, and each category and meta-category also receives a count feature, producing 1,013 features after pruning. Paired with incident labels at the domain level and trained with a gradient-boosted tree classifier, these features produce a calibrated breach probability. The domain as identifier is the load-bearing move: it avoids the error-prone IP-to-organization attribution that limits scan-based methods, and the privacy-policy heuristic is used to filter the negative sample set toward sites that plausibly belong to organizations.","core_discovery":"Using the domain name as the canonical identifier, the authors crawl each organization's website and record digital signatures of the technologies it runs. They pair these with positive labels from two cyber-incident corpora—general reported incidents and ransomware incidents—and negative labels from randomly sampled domains, then train a gradient-boosted tree classifier on 1,013 binary and count features. The central claim is that this domain-level model achieves strong ranking accuracy: area under the ROC curve of 92.3% for general incidents, 86.7% for ransomware incidents, and 89.2% when the corpora are combined, with a crawl yield above 95% on incident-report domains and 86.7% AUC on a 2024 ransomware set withheld from training. The authors interpret the cross-dataset drops (e.g., 79.3% when a general-incident-trained model scores ransomware victims) as evidence that ransomware victims differ systematically from the broader incident population, and they show that adding a coarse industry-sector feature improves accuracy on top of technology signatures.","pith_inferences":["Editorial inference: a deployed version could monitor risk continuously by re-crawling domains on a schedule, turning the point-in-time classifier into an early-warning system that flags sudden changes in technology stack as risk-relevant events; the paper does not develop this operationalization.","Editorial inference: the crawl features are genre-neutral enough that they could be tested against incident types outside the paper's labels (e.g., insider misuse, physical incidents, supply-chain compromise), since the model's positive set is limited to malware, hacking, social engineering, and ransomware.","Editorial inference: because the negative set requires a found privacy policy, the comparison may be selecting for organizations with certain regulatory or consumer-facing characteristics, so a sensitivity analysis without that filter would reveal how much of the signal is organizational form rather than security posture.","Editorial inference: the pool mismatch between breach victims and the random-domain negatives means the calibrated probability is safest used as a peer-relative ranking within sector and size bands, not as an absolute probability of breach for any given organization."],"forward_implications":["A risk score can be produced for any organization whose website is crawlable, including the millions of small and medium enterprises that lack dedicated, attributable IP address space; the paper reports a greater-than-95% data-collection yield on incident-report domains.","The model's probability output is well-calibrated without post-processing, so the scores can be interpreted directly as breach likelihood under the paper's label definitions.","Adding a twenty-category industry-sector feature reduces error by more than 40% relative to sector-only models, suggesting that cheap auxiliary data can be layered onto the crawl features.","Because general incidents and ransomware incidents produce different feature importances and cross-dataset AUCs, deploying a score for a specific use case requires training on the matching incident type.","A historically grounded prediction setup is only partially supported by archived snapshots: coverage drops and fewer technologies are detected, so the authors state that live crawls accumulated over time—not archive replay—are the route to true forecasting."],"supporting_citations":[{"why":"Prior IP-scan-based breach-forecasting model that this paper's domain-level approach is designed to replace; supplies the motivation and the baseline to beat.","marker":"[24]"},{"why":"Supplies the general cybersecurity-incident reports used as positive training and validation labels.","marker":"[44]"},{"why":"Supplies the ransomware incident reports used as positive labels and the 2024 held-out evaluation set.","marker":"[4]"},{"why":"Supplies the random list of top-ranked domains from which the non-incident comparison samples were drawn.","marker":"[28]"},{"why":"The source the paper cites in Section 6.2 for the known pool-difference limitation between breach-report positives and the broader negative pool.","marker":"[46]"},{"why":"Provides the gradient-boosted tree algorithm used to train the classifier on the tabular feature set.","marker":"[6]"}],"fun_headline_variants":["Website tech fingerprints predict data breaches for SMEs","Crawling public sites yields cyber risk scores at 92% AUC","SME cyber risk scored without IP scans via site signatures","Tech signatures on websites gauge breach odds with high AUC"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the randomly sampled, privacy-policy-hosting comparison domains are a valid stand-in for organizations that have not been breached; the paper itself concedes in Section 6.2 that the breached and non-breached pools differ systematically in size and sector, so if the model is mostly separating large organizations from small ones, its absolute probability estimates are not supported.","fun_headline_variants_meta":{"raw":{"variants":["Website tech fingerprints predict data breaches for SMEs","Crawling public sites yields cyber risk scores at 92% AUC","SME cyber risk scored without IP scans via site signatures","Tech signatures on websites gauge breach odds with high AUC"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000262,"raw_usage":{"total_tokens":1577,"prompt_tokens":904,"completion_tokens":673,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":520,"completion_tokens_details":{"reasoning_tokens":607}},"tokens_in":520,"tokens_out":673,"duration_ms":7603,"temperature":1.0,"reasoning_tokens":607,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:52:31.883057+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct a matching test set in which each breached organization is paired with a non-breached organization of the same sector and comparable size, and recompute the model's AUC: if discrimination falls toward chance on these matched pairs, the score is capturing demographic sorting rather than the security posture claimed for the technology signatures.","supporting_citations":[{"cited_title":"Cloudy with a chance of breach: Forecasting cyber security incidents","cited_arxiv_id":null,"evidence_quote":"Prior IP-scan-based breach-forecasting model that this paper's domain-level approach is designed to replace; supplies the motivation and the baseline to beat."},{"cited_title":"SoK: Quantifying cyber risk","cited_arxiv_id":null,"evidence_quote":"The source the paper cites in Section 6.2 for the known pool-difference limitation between breach-report positives and the broader negative pool."},{"cited_title":"XGBoost: A scalable tree boosting system","cited_arxiv_id":null,"evidence_quote":"Provides the gradient-boosted tree algorithm used to train the classifier on the tabular feature set."}],"review_version":1}