{"id":"2203e9b2-96ea-40c2-a9c4-42f3b2702c5f","arxiv_id":"2411.17931","paper_version":4,"verdict":"REJECT","confidence":"HIGH","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"A case study links dark web forum keyword counts to Shodan-discovered IoT devices, but does not validate any predictive model.","lead":"This paper describes a hand-built workflow: scan a small set of dark web forums for posts about hacking internet-connected devices, then use the Shodan search engine to find exposed sensors. The authors present this as a way to predict cyber attacks, but the paper contains no actual prediction and no test of predictive accuracy.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The case study never links the 11 forum posts to the Shodan results, so the central predictive correlation is asserted rather than demonstrated.","rationale":"The paper's central claim is a predictive threat-intelligence correlation between dark web discussions and Shodan-derived IoT vulnerabilities. The case study does assemble two collections: a handful of forum posts mentioning IoT hacking, and a Shodan search for 'sensor' that returns exposed devices. But the argument never connects these collections. There is no rule mapping forum content to Shodan queries, no timestamps to establish that discussion precedes compromise, and no attack ground truth to evaluate whether the predicted targets are actually attacked. The reader's weakest_assumption identifies exactly this gap: the unstated premise that keyword mentions in selected forums are a valid signal and that Shodan hits for 'sensor' correspond to the devices discussed. I agree with that assessment. The paper contains other inconsistencies, such as the 23-versus-22 website count in Table I, duplicated filtering text in Sections IV.B and IV.C.1, and the missing Figure 4, but these are secondary. The decisive issue is that the central predictive claim is unsupported by the evidence as presented. I would leave the REJECT verdict unchanged; the work could be reframed as a preliminary qualitative case study, but as submitted it does not substantiate the claimed correlation or the predictive framework.","tokens_in":7020,"tokens_out":2625,"duration_ms":25241,"concrete_test":"Build a temporal linkage test around the 11 posts: record each post's timestamp and exact device/vulnerability keywords; for each keyword, query Shodan or Censys historical scan data to enumerate exposed devices matching that keyword; then compare compromise rates (e.g., from honeypot feeds or Shodan vulnerability notifications) for those devices against matched exposed devices whose keywords were not discussed in the forums, over a fixed window such as 90 days after the post. If forum-discussed devices show no significant excess compromise after controlling for exposure, the claimed predictive correlation is refuted. This directly tests the missing link between Table II and Section IV.C.2.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing step in Section III.A is the claim that 'our method correlates hacker discussions ... with Shodan-derived device vulnerabilities to prioritize high-risk targets.' The case study does not perform that correlation. Section IV.C.1 reports only 11 forum posts across HackHound, Hackers Tribe, School-of-HackNet, and HackerWeb (Table II), with no timestamps, no target device identifiers, and no extraction of the specific exploits being discussed. Section IV.C.2 then runs a single Shodan API search for the keyword 'sensor' and reports 582 results, and the paper manually accesses a few devices on port 8080. Nothing ties those Shodan hits to the forums: the query 'sensor' is not derived from the forum text, no linkage rule is stated, and no temporal ordering is established. Consequently, the central claim rests on the unstated assumption that co-occurrence of two hand-collected datasets in one report is evidence of predictive correlation. For a method whose stated contribution is prediction, there is no baseline, no ground-truth attack data, and no statistical test, so the claimed correlation cannot be distinguished from coincidence. This is a missing-evidence failure of the central argument, not merely an underpowered sample.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a threat intelligence framework that combines dark-web forum data collection with Shodan device scanning to predict or prioritize cyber attacks against IoT devices. The methodology in Section III.A is a multi-stage pipeline (collection, back-link search, people search, meta-searching, filtering, analysis). The case study in Section IV applies a filtered version of the pipeline: 23 websites are manually selected, four forums are searched for IoT-related keywords, and a single Shodan search for the keyword 'sensor' returns 582 devices. The paper concludes that botnets, malware, and sensors are the most discussed IoT exploitation methods and that vulnerable devices can be directly accessed.","tokens_in":7140,"tokens_out":4797,"duration_ms":40554,"significance":"If validated, the claimed framework would be a useful addition to IoT threat intelligence. The manuscript has two concrete strengths: the methodology flowchart is clear, and the Shodan API script in Listing 1 is reproducible. However, the paper does not deliver what its title and abstract promise. No prediction is made or evaluated, no baseline or ground-truth attack labels are provided, and the alleged correlation between forum discussions and Shodan-discovered devices is never demonstrated. At present the contribution is an exploratory description of a data-collection pipeline, not a validated predictive method.","major_comments":[{"comment":"The central claim of the paper is prediction, but Section IV contains no prediction task, no forecast horizon, no temporal train/test split, no ground-truth attack data, no baseline, and no evaluation metric. The quantitative evidence consists of 11 forum posts in Table II and 582 Shodan results in Section IV.C.2. Without any evaluation, the title's 'Predict Cyber Attacks' and Section III.A's 'real-time cyber-attack prediction' are unsupported.","section":"IV.C"},{"comment":"The core correlation asserted in Section III.A is never established. The Shodan query is the single keyword 'sensor' (Listing 1), which does not come from the forum keyword sets in Table II (e.g., 'Botnet', 'Malware', 'Rats for Android devices'). No linkage rule, no device identifiers, and no timestamps connect the 11 forum posts to the 582 Shodan results. The case study therefore cannot support the claim that the method 'correlates hacker discussions ... with Shodan-derived device vulnerabilities to prioritize high-risk targets.'","section":"IV.C.1 and IV.C.2"},{"comment":"The data filtering is too subjective and underdocumented to support the conclusions. The reduction from hundreds of sources to 32 and then 23 websites is done by manual exclusion based on 'domain expertise' and language coverage; the NLP/ML threat relevance score is mentioned but its training set, feature details, and validation are not described. Because the final analysis rests on 11 posts from four forums, the lack of reproducibility of the filtering step is a load-bearing weakness rather than a minor detail.","section":"IV.B"},{"comment":"The descriptive statistics are internally inconsistent. The text says Figure 2 displays the percentage of IoT-related discussions in 'the seven forums we selected for analysis,' but Table II lists only four forums and reports a total of 11 posts. The figure's percentages cannot be checked against the table, and no counts are given for the other three purported forums. This inconsistency undermines confidence in the one quantitative result of the case study.","section":"IV.C.1"}],"minor_comments":[{"comment":"The Data Availability statement refers to 'Figure 4 (anonymized IoT device screenshots),' but the manuscript contains only Figures 1-3; the screenshots are in Figure 3.","section":"VI"},{"comment":"References [17] and [18] are the identical paper (Roman, Zhou, and Lopez, 'On the features and challenges of security and privacy in distributed internet of things'); one should be removed or replaced.","section":"References"},{"comment":"The affiliation 'Seattle, W A' contains a spacing typo and should read 'WA'.","section":"Author affiliations"},{"comment":"The Hansa Market example in Section II.A is not connected to the IoT case study; either link it to the analysis or omit it to keep the literature review focused.","section":"II.A"}],"recommendation":"reject","confidential_remarks":"The gap between the paper's framing and its evidence is too large for a minor or major revision to close without new data collection and evaluation. The authors should be encouraged to resubmit a substantially revised version that contains a real prediction experiment with baseline, ground truth, and temporal validation. I see no ethical or scope concerns beyond the lack of authorization discussion when accessing live devices."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: this one promises prediction but delivers a descriptive case study. The load-bearing claim—that dark web chatter can be correlated with Shodan exposure to predict attacks—is never demonstrated. The authors assemble a tiny, hand-curated dataset of 11 forum posts and a Shodan search for the literal keyword \"sensor\", but never connect the two. No temporal ordering, no linkage rule, no baseline, no ground truth.\n\nWhat's actually new: very little. The combination of dark web crawling, TF-IDF scoring, logistic regression, and Shodan enumeration is a composition of known techniques, and the authors cite similar prior work [13,14,21,25]. The case study does include real observations: a few forums do discuss IoT hacking, and Shodan returns exposed sensor devices with admin panels. That is mildly interesting but not novel.\n\nCredit where due: they were transparent about the manual filtering and the small scale, they published their code snippet, and their literature review is adequate. The paper is coherent in structure; the problem is evidentiary, not organizational.\n\nSoft spots, in order of severity. First and fatal: the claimed correlation is missing. Section III.A states the method \"correlates hacker discussions... with Shodan-derived device vulnerabilities,\" but Section IV.C reports forum post counts and Shodan results side by side with no link. The forum posts have no timestamps, device names, or exploit details. The Shodan query for \"sensor\" is generic, not derived from the forums. Consequently, the paper does not predict anything. Second: internal inconsistencies—the abstract says 23 websites, Table I lists 22; filtering text is duplicated almost verbatim in IV.B and IV.C.1; the Declaration references a Figure 4 that does not exist. These are minor on their own but reinforce a lack of polish. Third: no evaluation. Even if we read the paper as a qualitative case study, there is no attempt to compare against a baseline or external data.\n\nThe paper could be reframed as a preliminary observational study of IoT-related chatter on dark web forums, with explicit caveats that no predictive claim is made. As it stands, the title and abstract overclaim.\n\nWho would benefit: a methods instructor looking for examples of overclaiming, or a researcher comping IoT threat-intelligence papers for background. It does not deserve referee time in its current form.\n\nRecommendation: desk reject. If given a second chance, require the authors to either perform a real correlation or drop the prediction language.","headline":"Promises prediction but delivers an unconnected case study: the claimed dark-web-to-Shodan correlation is never performed.","tokens_in":7754,"tokens_out":2970,"would_cite":false,"duration_ms":25624,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper proposes a predictive threat-intelligence pipeline that correlates dark-web hacker discussions with Shodan-scanned IoT devices to prioritize high-risk targets before attack.","keywords":["Internet of Things (IoT)","Dark Web","Threat intelligence","Shodan","IoT vulnerability scanning","Cyber attack prediction","Hacker forum analysis","Machine learning in cybersecurity"],"falsifier":"Compare the 11 forum posts against the 582 Shodan 'sensor' results: if no device model, port, or IP mentioned in the posts appears among the exposed devices, the claimed link between forum discussion and Shodan-identified vulnerability has not been demonstrated.","tokens_in":6751,"feed_emoji":"📡","tokens_out":7483,"duration_ms":63889,"temperature":0.7,"pith_summary":"The paper tries to establish that systematically collecting dark-web hacker discussions and combining them with Shodan scans of internet-connected devices can reveal which IoT devices are at high risk of cyber attack before an attack happens. The authors propose a semi-automated pipeline that gathers candidate websites, filters them for security relevance using manual review plus keyword tagging and a logistic-regression threat score, then correlates the exploitation themes discussed in hacker forums with exposed devices found by Shodan. If the correlation holds, defenders could use forum chatter as an early-warning signal and prioritize patching or monitoring of the devices attackers are most interested in. The case study applies the pipeline to 23 dark-web websites, finding that IoT-hacking discussion concentrates on botnets, malware, and sensors, while a Shodan search for the keyword 'sensor' returns 582 devices, several with accessible administrative panels.","feed_headline":"Hacker forum posts plus Shodan scans predict IoT attack priorities","feed_subtitle":"A method links dark-web hacking talk about botnets and sensors to exposed devices, flagging risky targets first.","key_machinery":"The machinery is the correlation step between two halves. On the discussion side, the framework uses web crawling, back-link searches, meta-searching, and manual filtering, then TF-IDF-based keyword tagging with a logistic-regression threat score to rank websites by security relevance. On the device side, Shodan—an internet-wide search engine that indexes banner data from exposed devices—provides a searchable inventory of IoT hardware, including company, IP, port, and operating system. The framework's predictive power is meant to live in the link between these two halves: a theme discussed in a hacker forum, such as botnets or sensors, becomes a Shodan query for exposed devices of that kind, and the overlap is treated as a high-risk target.","core_discovery":"On its own terms, the paper's central claim is that attacker discussions on dark-web forums are not random noise: they concentrate on recognizable IoT exploitation themes, and those same themes can be used to query Shodan and locate real, internet-facing devices that match them. Across the forums examined, three of four selected forums showed 12–30% of their topics discussing IoT hacking, with botnets, malware, and sensors the most-discussed methods. A Shodan search for 'sensor' returned 582 results, and manual inspection found exposed temperature and pressure sensors with open port 8080 and administrative panels reachable without SSL. The paper interprets this overlap as evidence that correlating dark-web discussion with Shodan exposure can prioritize high-risk IoT targets.","pith_inferences":["A testable extension is to time-align forum discussion spikes with subsequent attack telemetry against Shodan-identified devices; the paper's snapshot does not yet show temporal ordering, so this would measure how much early warning the correlation actually gives.","The same two-sided correlation could generalize beyond IoT to any attack surface where a textual discussion space and a device-inventory search engine both exist, such as webcams or industrial control systems.","The literal Shodan query 'sensor' could be replaced by an automated mapping from forum keywords to Shodan filters, which would let the framework be scored against historical attack data and reveal which keyword-to-device mappings are predictive."],"forward_implications":["Security teams could watch dark-web forums for emerging IoT exploit keywords and immediately run those keywords through Shodan to build a shortlist of exposed devices to patch or monitor first.","The pipeline offers a reusable template for semi-automated threat intelligence: crawl candidate sites, filter with a mix of human review and machine scoring, then scan for matching vulnerable devices.","The case study implies that mundane devices such as temperature and pressure sensors should be treated as high-value targets when they appear in forum discussions and are internet-accessible.","For IoT manufacturers, the results support the case for strict security standards, dedicated security-conscious software development, and user best practices as the paper's conclusion recommends."],"supporting_citations":[{"why":"It establishes that Shodan indexes newly connected internet-facing devices within 14 days, grounding Shodan as a device-discovery tool.","marker":"[1]"},{"why":"It provides an earlier dark-web case study that this paper's collection and analysis methodology extends.","marker":"[13]"},{"why":"It supplies the topology-oriented view of dark networks that underlies the methodology's information-source and analysis steps.","marker":"[14]"},{"why":"It supports the use of dark-web forum analysis and manual filtering techniques for threat intelligence.","marker":"[23]"},{"why":"It serves as the prior approach the paper explicitly contrasts, arguing that correlating hacker discussions with Shodan vulnerabilities goes beyond it.","marker":"[25]"}],"fun_headline_variants":["Dark web chatter plus Shodan scans predict IoT threats","Linking hacker forums to exposed devices forecasts attacks","Dark web IoT talk matches vulnerable sensors online","Predicting cyber attacks from dark web and Shodan data"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that keyword mentions in eleven posts across four manually selected dark-web forums signal real-world attacker targeting, and that Shodan results for the literal keyword 'sensor' correspond to the devices those posts describe.","fun_headline_variants_meta":{"raw":{"variants":["Dark web chatter plus Shodan scans predict IoT threats","Linking hacker forums to exposed devices forecasts attacks","Dark web IoT talk matches vulnerable sensors online","Predicting cyber attacks from dark web and Shodan data"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000491,"raw_usage":{"total_tokens":2386,"prompt_tokens":888,"completion_tokens":1498,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":504,"completion_tokens_details":{"reasoning_tokens":1447}},"tokens_in":504,"tokens_out":1498,"duration_ms":9495,"temperature":1.0,"reasoning_tokens":1447,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T11:41:13.756258+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compare the 11 forum posts against the 582 Shodan 'sensor' results: if no device model, port, or IP mentioned in the posts appears among the exposed devices, the claimed link between forum discussion and Shodan-identified vulnerability has not been demonstrated.","supporting_citations":[{"cited_title":"”Impact of the Shodan computer search engine on internet-facing industrial control system devices.” No","cited_arxiv_id":null,"evidence_quote":"It establishes that Shodan indexes newly connected internet-facing devices within 14 days, grounding Shodan as a device-discovery tool."},{"cited_title":"”Uncovering the dark Web: A case study of Jihad on the Web.” Journal of the American Society for Information Science and Technology 59.8 (2008): 1347-1359","cited_arxiv_id":null,"evidence_quote":"It provides an earlier dark-web case study that this paper's collection and analysis methodology extends."},{"cited_title":"”The topology of dark networks.” Communications of the ACM 51.10 (2008): 58-65","cited_arxiv_id":null,"evidence_quote":"It supplies the topology-oriented view of dark networks that underlies the methodology's information-source and analysis steps."},{"cited_title":"”Affect intensity analysis of dark web forums.” Intelligence and Security Informatics, 2007 IEEE","cited_arxiv_id":null,"evidence_quote":"It supports the use of dark-web forum analysis and manual filtering techniques for threat intelligence."},{"cited_title":"”IoT Threat Report.” https://iotbusinessnews.com/ download/white-papers/UNIT42-IoT-Threat-Report.pdf, 2020","cited_arxiv_id":null,"evidence_quote":"It serves as the prior approach the paper explicitly contrasts, arguing that correlating hacker discussions with Shodan vulnerabilities goes beyond it."}],"review_version":1}