{"id":"c774ade7-2daf-484a-927c-b7593a08a50e","arxiv_id":"2411.12435","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"A socio-technical model combining network misconfiguration scans and Twitter engagement features reports 98.43% AUC for classifying US organizations with reported hacking breaches, 12 points above technical features alone, though the setup is not a clean temporal prediction.","lead":"This paper builds machine learning models that combine technical security scans and Twitter signals to estimate whether US organizations will suffer a hacking breach. It reports AUC above 98% when social features are added, a 12-point gain over technical features alone, but the evaluation has temporal and label-correction issues that weaken the predictive claim.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 98% AUC is computed on a same-window random split, so features aggregated through September 2019 can include post-breach signals for organizations breached earlier in the 2016-2019 period; the headline metric is not demonstrated to be predictive.","rationale":"The reader's weakest assumption correctly identifies the most load-bearing flaw: the random-split evaluation on aggregated 2016-2019 data cannot establish predictive performance because features for test victims may include post-breach information. This directly undermines the abstract's central claim of a predictive system with AUC exceeding 98%. The label-correction step compounds the issue by making test labels partly dependent on models trained on the full dataset. The forecasting experiment, while directionally better, is too small and too weakly reported to rescue the claim. I agree with the reader's REJECT verdict; no adjustment to the verdict is needed. The proposed chronological-split test would settle whether the concern lands by measuring AUC under a protocol that excludes post-breach feature contamination and model-generated test labels.","tokens_in":22054,"tokens_out":2383,"duration_ms":26351,"concrete_test":"Re-run the main classification with a strictly chronological split: train only on organizations whose incidents (or, for non-victims, whose inclusion date) occur before a cutoff, using features aggregated only over the period before that cutoff; test on organizations with incidents after the cutoff using only features measured in the 6 or 12 months prior to each incident. Report AUC with bootstrap confidence intervals and compare against the technical-only baseline. If the AUC drops substantially below 98% or the 12% social-feature gain shrinks, the claimed predictive performance is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that socio-technical profiles predict whether an organization will experience a hacking breach, with AUC exceeding 98%. The evaluation in Section VI.B uses a random 70/30 train/test split on a dataset whose features and labels both span January 2016 to September 2019 (Section III). This permits lookahead: for any victim organization whose incident occurred early in the window, the aggregated technical and Twitter features (e.g., open ports, expired certificates, likes ratio, retweet ratio) include information measured after the breach. The model can therefore learn post-breach correlates rather than pre-breach risk. The paper's own limitation statement in Section VIII acknowledges that 'STRisk is trained on aggregated data over the considered period.' Additionally, the label-correction step in Section V.C flips 347 negative labels using out-of-sample predictions from models fit on the full dataset before the train/test split, so the test labels are partly model-generated; this can further inflate the reported AUC. The forecasting experiment in Section VII.B uses features measured before each incident, but it is very small, reports only accuracy (42.85% for 3 months, 71.42% for 6 months), and does not provide AUC or confidence intervals, so it cannot substitute for a proper temporal evaluation of the headline claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes STRisk, a supervised learning system that combines externally measurable technical indicators (open ports, blacklists, darknet sightings, SSL certificates, spam domains) with Twitter-derived social signals (volume, retweet/like ratios, sentiment) to classify US organizations as likely or unlikely to experience a hacking breach. The dataset contains about 3,800 organizations, with reported incidents from PRC and VCDB between January 2016 and September 2019 and an ARIN-sampled negative set. The authors introduce a noise-correction procedure that flips some negative labels to positive based on out-of-sample predictions from six models, then train and evaluate tree-based, linear, and stacked classifiers. They report an AUC exceeding 98% for the combined socio-technical features, about 12% higher than technical features alone, and offer a SHAP-based feature importance analysis. A smaller forecasting experiment using features measured before incidents from March 2020 to March 2021 reports accuracies of 42.85% (3 months) and 71.42% (6 months).","tokens_in":22286,"tokens_out":6517,"duration_ms":58987,"significance":"If validated, STRisk would be an interesting demonstration that externally observable technical and social signals can contribute to organizational cyber-risk assessment, and the explicit treatment of unreported breaches is a useful direction. The data collection is substantial, and the feature-importance analysis with SHAP is a strength. However, the headline result is not supported by the current evaluation protocol: the random split and the use of model-generated flipped labels allow lookahead and circularity, so the 98% AUC cannot be interpreted as predictive skill. The forecasting experiment, while correctly using prior data, is far too small to compensate. The contribution is therefore plausible but unproven, and requires a substantially revised evaluation.","major_comments":[{"comment":"The evaluation does not enforce a temporal ordering between features and incidents. Both the socio-technical profiles and the labels are aggregated over January 2016 to September 2019 (Section III), and the 70/30 split in Section VI.B is random. Consequently, for any organization whose breach occurred early in that window, the aggregated features include measurements taken after the breach (e.g., open ports, expired certificates, tweets posted later in the period). The model can therefore learn post-breach correlates, and the AUC values in Table XII are contemporaneous association metrics rather than predictive performance. The paper's own limitation statement in Section VIII ('STRisk is trained on aggregated data over the considered period') acknowledges this issue.","section":"Section VI.B and Section III"},{"comment":"The label-correction step in Section V.C flips 347 negative examples to class 1 using predictions from six models, and the final classifiers in Section VI.B are trained and tested on the corrected dataset. Because the six models are fit on the full dataset (including the test portion) before the split, the test labels are partly generated by models that have already seen the test features. No nested cross-validation or separate holdout is described to prevent this circularity. This can substantially inflate the reported AUC in Table XII. The authors should either evaluate on a holdout set whose labels are never modified by the correction procedure, or apply noise correction only within training folds.","section":"Section V.C and Section VI.B"},{"comment":"The forecasting experiment cannot substitute for a proper temporal evaluation. It uses only 7 incidents (the reported accuracies 42.85% and 71.42% correspond to 3 and 5 correct predictions, respectively), reports only accuracy with no AUC, confidence intervals, or baseline comparison, and uses only victim organizations. With such a small sample, the difference between the 3-month and 6-month results is not statistically meaningful. This experiment does not support the statement that STRisk 'performs well in predicting future data breaches.'","section":"Section VII.B"},{"comment":"The validation of the noise-detection method is performed on artificial noise injected into the positive sample, not on the actual negative sample where the correction is applied. The reported mean detection accuracy (e.g., 82.1% for all models combined) may not transfer to the negative sample, which has a different class distribution and feature characteristics. Since the noise correction changes 347 labels (Table IX), the validity of this estimate directly affects the reliability of the corrected dataset.","section":"Section V.B"}],"minor_comments":[{"comment":"The abstract says 'a Area Under Curve'; it should read 'an Area Under the Curve'.","section":"Abstract"},{"comment":"The sentiment neutrality condition is written as '−0.1 <= p <= −0.1'; it should be '−0.1 <= p <= 0.1'.","section":"Section IV.B"},{"comment":"The 'Org Size' column appears misaligned: the mean is 1.600 while the standard deviation is 414912 and the maximum is 22404, which is internally inconsistent; the column likely contains different statistics or a formatting error.","section":"Table VI"},{"comment":"The word 'Satcking' is a typo for 'Stacking' in two rows of the table.","section":"Table XII"},{"comment":"The label 'T echnical' should be 'Technical'.","section":"Table XIII"},{"comment":"The text says 'the feature importance returned by catboost' but then discusses XGBoost; please clarify which model's importances are presented in Table XV and Fig. 5.","section":"Section VII.C"},{"comment":"Reference [44] (Confident Learning) lacks volume, issue, and page numbers.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within the journal's scope, but the evaluation protocol is the main obstacle. I recommend inviting a major revision with clear instructions to redo the evaluation with a temporal split and a label-correction protocol that cannot contaminate the test set. The forecasting experiment should also be expanded or removed as evidence for the headline claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Peer, here's the short version. The paper's central claim—that socio-technical profiles predict hacking breaches with 98% AUC—is not supported by the evaluation as run. Features and breach labels both cover Jan 2016–Sep 2019, and the train/test split is random after label correction, so aggregated features can include post-breach signals. The label-correction step flips 347 negatives using predictions from models fit on the full dataset before the split, which makes part of the test labels model-generated. The authors acknowledge in Section VIII that STRisk is trained on aggregated data. The forecasting experiment in VII.B is too small (looks like 7 incidents) and reports only accuracy, so it can't rescue the headline.\n\nThat said, the paper is not a throwaway. The configuration is genuinely new relative to the cited work: organization-level breach prediction combining external technical scans, Twitter engagement and sentiment, and confident-learning-style correction for unreported incidents. The data collection is real work—matching PRC and VCDB records to ARIN IP ranges with manual verification, scraping 2.5M tweets, building the feature set. The feature importance analysis is plausible and produces interpretable claims about open ports, expired certificates, likes ratio, and retweet ratio. And the limitations section is honest; they flag the aggregated-data issue themselves.\n\nThe soft spots are proportional. The temporal leak is load-bearing, not cosmetic, for the stated predictive claim. The label-correction step compounds it. There's no code or data, so verification isn't possible. But the core research question—do external technical and Twitter signals add predictive value for breach risk?—is important and the paper gives a reasonable template for how to test it. The fix is known: temporal split, nested label correction, and a real forecasting holdout.\n\nI'd send this to peer review rather than desk reject, with the expectation of major revision. A good referee could push the authors to redo the evaluation properly. If they can't or won't, reject. For your own reading, treat the 98% and the 12-point gain as hypotheses, not results. This is for cyber-risk researchers and insurance-modeling folks, as a cautionary example and a source of feature ideas.","headline":"The 98% AUC headline is a same-window classification score with label leakage, not a predictive result; still, the socio-technical feature combination and data effort deserve a fair shot under a temporal evaluation.","tokens_in":22849,"tokens_out":2609,"would_cite":false,"duration_ms":25298,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Combining external network scans with Twitter engagement separates breached from non-breached US organizations with an AUC above 98%, 12 points higher than technical features alone.","keywords":["data breach prediction","hacking breach risk assessment","socio-technical features","Twitter social signals","network misconfiguration","noisy labels","confident learning","machine learning classification"],"falsifier":"Strictly temporal validation would settle the claim: train STRisk only on organizations and features observed before a cutoff, measure test features only up to each test organization's breach or disclosure date, and compare AUC against the reported 98%. The paper's own forecasting-mode test reports 42.85% accuracy with three months of prior data and 71.42% with six months; re-running under a strict temporal split would show whether the 98% AUC survives without same-period aggregation.","tokens_in":21827,"feed_emoji":"🛡️","tokens_out":6726,"duration_ms":59179,"temperature":0.7,"pith_summary":"STRisk aims to show that an organization's risk of a hacking breach can be predicted from outside, using a combination of technical signals (open ports, expired certificates, blacklisted and darknet IPs, spam domains) and social signals from Twitter (mentions, retweets, likes, sentiment). On more than 3,800 US organizations, the paper reports an AUC above 98% when both signal types are used, about 12 points higher than technical features alone. The paper also treats non-victim labels as noisy, since many breaches go unreported, and flips detected mislabeled organizations from class 0 to class 1 before training. If correct, this gives insurers and security teams a cheap, externally measurable risk score that improves on prior technical-only assessments. The strongest technical predictors are open ports and expired certificates, while the strongest social predictors are spreadability and agreeability.","feed_headline":"Twitter signals lift breach forecasts past 98% AUC","feed_subtitle":"External network scans plus social reputation beat technical indicators alone by 12 points on 3,800 US organizations.","key_machinery":"The load-bearing object is the per-organization socio-technical profile, a feature vector that pairs externally measured network misconfigurations (number and ratio of open ports, expired certificates, blacklisted IPs, darknet IPs, and spam domains) with Twitter-derived engagement and sentiment statistics (mentions, unique accounts, retweet and like ratios, and polarity counts). In the paper's naming, spreadability is retweets per tweet, agreeability is likes per tweet, and debatability is retweets per reply. The second piece of machinery is the label-correction step: a noise transition matrix estimated by confident learning from out-of-sample predictions of six classifiers, which identifies non-victim organizations whose profiles look like victims and flips their labels before supervised training. The profile carries the prediction; the label correction removes the main known bias in the training set.","core_discovery":"The central claim is that a socio-technical profile constructed entirely from external observations separates breached from non-breached organizations on the studied sample with an area under the ROC curve above 98%. XGBoost and CatBoost reach AUCs of 98.43% and 98.31% using technical features, Twitter features, sector, and organization size, versus roughly 86% with technical features alone; the paper interprets the gap as evidence that social posture carries independent predictive information. A secondary claim is that the negative sample is noisy, and that a confident-learning procedure over six classifiers can identify and flip 347 of 3,016 supposed non-victims, reducing label noise before training. The paper additionally reports that in a forecasting-style test with features measured only before new incidents from March 2020 to March 2021, accuracy rises from 42.85% with three months of prior data to 71.42% with six months, suggesting the model benefits from a longer pre-incident history.","pith_inferences":["A natural stress test is to rebuild the profiles with a strict temporal lag, using only data from before each organization's breach date; if the 12-point social gain disappears, the same-period aggregation rather than social signals drove the result.","The same socio-technical profile could be transferred to other reputational channels such as LinkedIn, Reddit, or dark-web forums to see whether spreadability reflects genuine exposure or just Twitter-specific attention.","If the noise-correction step is re-run with only pre-breach features, the 347 flipped labels could shrink; comparing the two flipped sets would quantify how much leakage the confident-learning procedure absorbs.","For insurers, the practical extension is a rolling risk score: retrain monthly on the trailing 12 months of external signals and test on the next month's publicly disclosed breaches, which would convert the 98% AUC into a usable forward forecast."],"forward_implications":["Organizations can be ranked by breach risk using only public data: port scans, certificate checks, blacklists, and Twitter engagement, without any internal access.","Twitter-derived features contribute roughly 42% of model importance and, used alone, match the AUC of technical features alone, so social reputation is not a minor supplement.","The label-flipping procedure implies that around 11% of apparently breach-free US organizations in the sample share victim-like profiles, a measurable estimate of unreported incidents.","The forecasting-mode result suggests that prediction quality depends on the observation window; six months of accumulated socio-technical history beats three months."],"supporting_citations":[{"why":"Supplies the external technical-assessment baseline that STRisk extends and outperforms.","marker":"[3]"},{"why":"Provides a business-profile breach prediction baseline that STRisk compares against.","marker":"[4]"},{"why":"Supplies the internal-measurement baseline (RiskTeller) that STRisk claims to beat by over 3% AUC.","marker":"[13]"},{"why":"Ground-truth source for publicly reported US data breaches of type HACK.","marker":"[29]"},{"why":"Second ground-truth source for malware and hacking incidents used to build the victim set.","marker":"[30]"},{"why":"Provides the TCP port and SSL certificate scans used to build technical features.","marker":"[36]"},{"why":"Provides the blacklist data for malicious-IP features.","marker":"[37]"},{"why":"Tool used to scrape the roughly 2.5 million tweets that feed the social features.","marker":"[40]"},{"why":"The confident-learning method that the paper adapts to discover and correct noisy negative labels.","marker":"[44]"}],"fun_headline_variants":["Social signals boost breach prediction to 98% AUC","Hack risk: social chatter + open ports beat tech alone","Noisy data corrected: 98% AUC for breach exposure","Predicting breaches: social and technical combo wins","98% AUC: social media plus scans spot breach risk"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes that features aggregated over January 2016 to September 2019 can be used to predict breaches from the same period without lookahead; if future events in a breach's own history leak into its profile, the 98% AUC measures contemporaneous correlation rather than predictive risk.","fun_headline_variants_meta":{"raw":{"variants":["Social signals boost breach prediction to 98% AUC","Hack risk: social chatter + open ports beat tech alone","Noisy data corrected: 98% AUC for breach exposure","Predicting breaches: social and technical combo wins","98% AUC: social media plus scans spot breach risk"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000234,"raw_usage":{"total_tokens":1500,"prompt_tokens":952,"completion_tokens":548,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":568,"completion_tokens_details":{"reasoning_tokens":468}},"tokens_in":568,"tokens_out":548,"duration_ms":5846,"temperature":1.0,"reasoning_tokens":468,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T17:32:24.231070+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Strictly temporal validation would settle the claim: train STRisk only on organizations and features observed before a cutoff, measure test features only up to each test organization's breach or disclosure date, and compare AUC against the reported 98%. The paper's own forecasting-mode test reports 42.85% accuracy with three months of prior data and 71.42% with six months; re-running under a strict temporal split would show whether the 98% AUC survives without same-period aggregation.","supporting_citations":[{"cited_title":"Cloudy with a chance of breach: Forecasting cyber se curity incidents,","cited_arxiv_id":null,"evidence_quote":"Supplies the external technical-assessment baseline that STRisk extends and outperforms."},{"cited_title":"Risky busin ess: Fine- grained data breach prediction using business profiles,","cited_arxiv_id":null,"evidence_quote":"Provides a business-profile breach prediction baseline that STRisk compares against."},{"cited_title":"Riskteller: Predi cting the risk of cyber incidents,","cited_arxiv_id":null,"evidence_quote":"Supplies the internal-measurement baseline (RiskTeller) that STRisk claims to beat by over 3% AUC."},{"cited_title":"Privacy rights clearinghouse,","cited_arxiv_id":null,"evidence_quote":"Ground-truth source for publicly reported US data breaches of type HACK."},{"cited_title":"V eris community database,","cited_arxiv_id":null,"evidence_quote":"Second ground-truth source for malware and hacking incidents used to build the victim set."},{"cited_title":"[Online]","cited_arxiv_id":null,"evidence_quote":"Provides the TCP port and SSL certificate scans used to build technical features."},{"cited_title":"Firehol ip lists,","cited_arxiv_id":null,"evidence_quote":"Provides the blacklist data for malicious-IP features."},{"cited_title":"twitterscraper","cited_arxiv_id":null,"evidence_quote":"Tool used to scrape the roughly 2.5 million tweets that feed the social features."},{"cited_title":"Confident le arning: Es- timating uncertainty in dataset labels,","cited_arxiv_id":null,"evidence_quote":"The confident-learning method that the paper adapts to discover and correct noisy negative labels."}],"review_version":1}