{"id":"6e04b092-850c-439f-a880-04e664905cdc","arxiv_id":"2412.06015","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A set-partitioned Isolation Forest variant detects synthetic network scan anomalies at IP level, with best results on unusual port-service pairings but modest precision on volume spikes.","lead":"siForest, a modified Isolation Forest that groups network scans by IP, stops splitting early and scores IPs as sets. It outdoes classic preprocessing on synthetic scans with unusual port-service pairs, but only modestly and without error bars.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'consistently good' claim is not statistically supported: only point estimates are reported and the min-score aggregation rule was chosen post hoc on the generated data.","rationale":"Reader's weakest assumption is that synthetic data resembles real Censys scans. I agree this is a real limitation, but I see a more immediate problem for the stated strongest claim: even on the synthetic data, the evidence for siForest's 'consistent' superiority is not established because the paper reports no uncertainty and selects the min aggregation rule after seeing the data. The Section 4.1 sentence about minimum score makes this explicit. Given the central claim is a comparative ranking, missing error bars alone would make it difficult to verify; adding a post hoc choice makes it fragile. The reader's conditional verdict already captures the broad concern, so I would not change the verdict, but the concrete test should be a statistical re-analysis and out-of-sample aggregation selection rather than only real-data validation. The internal algorithm description is plausible and the implementation appears re-implementable, so no objection to the method's basic soundness.","tokens_in":6894,"tokens_out":6085,"duration_ms":63396,"concrete_test":"From the released or re-implemented siForest code, rerun all 10 random seeds for both anomaly types with both min and average IP-score aggregation, and report per-seed F2 plus 95% bootstrap confidence intervals for each method. Then perform a split: choose the aggregation rule on seeds 1–5 and evaluate on seeds 6–10. If siForest is not the only method whose held-out F2 exceeds the chosen threshold on both anomaly types, or if the min-vs-average gap reverses, the 'only consistently good' conclusion fails. As a secondary check, run the same pipeline on a real labeled Censys snapshot to test transfer of the ranking.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim—that siForest is the only method with consistently good performance—depends on the comparison in Section 4.4. That comparison is under-reported as it stands: metrics are single precision/recall/F2 values with no standard deviation, confidence interval, or per-seed breakdown across the 10 random datasets mentioned in Section 4.1. More importantly, Section 4.1 states that IP-level scores use the minimum aggregation option 'as it gives a better performance for the generated data,' i.e., the evaluation criterion was used to select the aggregation rule. With only two anomaly types, choosing the rule that looks best on the same data can inflate the apparent advantage of siForest. Consequently, the observed margins (e.g., type-1 F2 0.435 vs. summarization's 0.59; type-2 F2 0.599 vs. flattening's 0.344) do not establish that siForest is uniquely consistent; the ranking could shift under the average-score aggregator or when error bars are included. The lack of validation on real Censys data, acknowledged in Section 5, is a further external-validity limit, but the statistical issue undermines even the synthetic-data claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes siForest, a modification of the Isolation Forest algorithm for set-structured network scan data. The key idea is to stop tree partitioning when all data points in a node belong to the same IP address, thereby preserving the contextual relationship between ports and services while operating at the IP level. Two IP-level scoring aggregation strategies (minimum and average) are introduced. The algorithm is compared against two iForest baselines using different preprocessing approaches (flattening and summarization) on synthetic datasets that mimic Censys internet scans, with two anomaly types: usage spikes (type 1) and non-standard service-port combinations (type 2). The paper reports precision, recall, and F2-score for each method and anomaly type, and concludes that siForest is the only method showing consistently good performance across both anomaly types.","tokens_in":7105,"tokens_out":2678,"duration_ms":26579,"significance":"If the central claim were fully supported, siForest would be a practically useful extension of Isolation Forest for IP-level anomaly detection on network scan data, because it retains the low feature dimensionality of the flattened representation while producing IP-level predictions. The algorithmic modification is simple, clearly described, and plausible, and the comparison against two natural preprocessing baselines on identically generated data is a reasonable starting point. However, the significance is currently limited by the evaluation: the experiments are entirely synthetic, no variance or significance measures are reported, the best-performing aggregation rule appears to have been selected post hoc on the same data, and the reported type-1 precision for siForest (0.285) is low. These issues mean that the paper's main claim is not yet established, although the work is a promising direction that could be strengthened with additional experiments and statistical reporting.","major_comments":[{"comment":"The paper states in Section 4.1 that 10 different random datasets are generated for each anomaly type, but Section 4.4 reports only single precision, recall, and F2 values for each method, with no standard deviations, confidence intervals, per-seed results, or significance tests. Without this information, the claim that siForest is 'the only methods among the tested ones that shows consistently good performance' (Section 5) is not statistically supported. The authors should report per-dataset results and either paired statistical tests or at least variance measures across the 10 runs, so that the reader can assess whether the observed differences are stable.","section":"Section 4.1 and Section 4.4"},{"comment":"The authors write: 'We calculate anomaly scores at the IP level using the minimum score option as it gives a better performance for the generated data.' This sentence indicates that the aggregation rule was selected based on its performance on the same generated data used for evaluation. With only two anomaly types, such post hoc selection can inflate the apparent advantage of siForest, since the reported metrics are the result of choosing the more favorable rule. To make the comparison fair, the authors should either justify the minimum aggregation rule on independent grounds, or report results for both the minimum and average aggregation strategies for both anomaly types, and discuss sensitivity.","section":"Section 4.1"},{"comment":"The claim that siForest 'comes close' to the summarization approach for type 1 anomalies is not well supported by the reported numbers. For type 1, siForest has precision 0.285, recall 0.520, and F2 0.435, while summarization has precision 0.61, recall 0.58, and F2 0.59. The F2 difference (0.435 vs. 0.59) is substantial, and the precision is less than half. The conclusion that siForest is 'consistently good' depends on a threshold of what counts as acceptable performance; the paper should either justify such a threshold, report a combined metric that accounts for precision and recall more transparently, or soften the conclusion to reflect that siForest is second-best on type 1 anomalies.","section":"Section 4.4, Figure 6, and Section 5"},{"comment":"The external validity of the evaluation is limited by the use of synthetic data. Section 4.2 states that normal network behavior is based on 'common service-port combinations documented in Censys,' but no validation is provided that the generated data distribution resembles real Censys scans in terms of frequencies, correlations, or anomaly characteristics. Section 5 acknowledges that testing on real-world datasets is future work, but this limitation is load-bearing for the paper's central claim: if the synthetic distribution differs from real network traffic, the performance rankings may not transfer. The authors should either validate the synthetic data against real scans (for example, by comparing marginal distributions or using a small labeled real dataset) or explicitly restrict the conclusion to the synthetic setting.","section":"Section 4.2 and Section 5"}],"minor_comments":[{"comment":"The section heading contains a typo: 'Experiemntal Setup' should be 'Experimental Setup.'","section":"Section 4.1"},{"comment":"The sentence 'It’s main advantages are the high computational efficiency' should use 'Its main advantages' instead of 'It’s main advantages.'","section":"Section 2"},{"comment":"The sentence 'siForest is the only methods among the tested ones' has a subject-verb agreement error and should read 'the only method.'","section":"Section 5"},{"comment":"The abstract says siForest has 'the potential to outperform traditional approaches on some types of internet scan data,' while Section 5 makes the stronger claim that it is the only method with consistently good performance. The strength of the language should be aligned with the actual evidence.","section":"Abstract and Section 5"},{"comment":"The two plots in Figure 6 are not accompanied by a description of their axes, legends, or error bars. Adding axis labels, method labels, and variance information would make the figure self-contained and easier to interpret.","section":"Figure 6"}],"recommendation":"major_revision","confidential_remarks":"The manuscript reads like an early-stage workshop paper. The core algorithmic idea is reasonable and the presentation is generally clear, but the experimental evidence for the main claim is under-reported. The lack of variance measures, the post hoc selection of the aggregation rule, and the absence of any real-data validation are the key issues that need to be addressed before the paper can support its conclusion. I would also encourage the editor to ask the authors to release code and synthetic data generation scripts, as the paper currently provides no reproducibility artifacts."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea in this paper is modest but clean: modify Isolation Forest so that it stops splitting when all the scans in a node belong to the same IP, then aggregate the resulting leaf scores per IP. That gives you an IP-level anomaly detector without flattening the scan lists or exploding the feature space with one-hot summaries. I think that is genuinely new and it is described clearly enough to reimplement. The two anomaly types—volume spikes and non-standard service-port pairs—are reasonable, and the paper is honest that recall matters more than precision for this use case.\n\nThe problem is the evidence for the headline claim. The paper says siForest is the only method with consistently good performance across both anomaly types. What we actually get is a single precision/recall/F2 point per method per anomaly type, with no standard deviation, no interval, no per-seed breakdown, and no significance test, even though the setup says 10 random datasets were generated for each type. On top of that, Section 4.1 admits the min-score aggregation rule was chosen because it gives better performance on the generated data. That is post hoc selection on the evaluation set, which biases the comparison in siForest's favor. And the numbers are not that convincing: for type 1, siForest has F2 0.435 versus summarization's 0.59, and its precision is 0.285. So \"consistent\" is doing a lot of work; it means siForest is mediocre at both types while each baseline is good at one type and bad at the other. That may actually be a useful practical property, but it is not the same as being consistently good, and without variance estimates we cannot even tell whether the relative ordering is stable across seeds.\n\nI agree with the stress-test note that this is the load-bearing weakness. The fix is straightforward: report per-seed results with error bars, pre-specify the aggregation rule or use a validation split to select it, and try at least one real scan dataset, even a small sample. The synthetic generator's resemblance to Censys is asserted but never checked, so external validity is unknown.\n\nThere are some minor issues as well: a typo in \"Experiemntal,\" a related-work section that reads more like a citation dump than a positioning, and the figures are basic. None of these are fatal.\n\nWho is this for? Someone working on network anomaly detection who wants a cheap IP-level baseline to compare against. I would not cite it for the empirical results, but I would cite it for the algorithm idea if I needed a set-structured iForest variant. The paper deserves a serious referee because the idea is simple, plausible, and testable; peer review could push it into a decent short paper. As is, I would not accept it without major revision.","headline":"A simple, clearly-described variant of iForest for IP-level scan data; the consistency claim is plausible but the reported evidence is too thin to support it.","tokens_in":7580,"tokens_out":3628,"would_cite":true,"duration_ms":36313,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"siForest, an Isolation Forest variant that stops splitting at IP boundaries, is the only tested method that detects both volume spikes and unusual port-service pairings consistently.","keywords":["siForest","Isolation Forest","set-structured data","network anomaly detection","internet scan data","service-port anomalies","unsupervised anomaly detection","attack surface identification"],"falsifier":"Run siForest and the two preprocessing baselines on labeled real internet scan data with known malicious or misconfigured IPs and compare per-IP recall within each anomaly class; if either baseline matches or beats siForest's recall on both usage spikes and non-standard port-service pairs, the consistency claim is falsified.","tokens_in":6697,"feed_emoji":"🛡️","tokens_out":7768,"duration_ms":70774,"temperature":0.7,"pith_summary":"The paper tries to establish that internet scan data is best treated as sets, with each IP holding its own list of ports and services, rather than as flattened rows or per-IP count vectors. It introduces siForest, a version of Isolation Forest whose trees stop splitting once a node contains scans from only one IP, then aggregate per-scan anomaly scores up to the IP level. On synthetic data modeled after internet-wide scans, siForest is the only one of the three tested methods that keeps reasonably strong precision, recall, and F2-score on both anomaly types: usage spikes and non-standard service-port combinations. If correct, this gives network defenders a low-dimensional, IP-level detector that does not require knowing the anomaly type in advance.","feed_headline":"Split-by-IP isolation forest is only method to catch both anomaly types","feed_subtitle":"siForest keeps each IP's port-service pairs intact, preserving context that flat or summarized baselines lose.","key_machinery":"The Set-Partitioned Isolation Forest (siForest) is an Isolation Forest whose tree construction stops splitting when all data points in a node belong to the same IP address, so each leaf is a set of scans from one IP. This contrasts with two preprocessing baselines: flattening, which expands each scan into IP-port-service rows and thus loses the IP grouping, and summarization, which creates one row per IP with counts for hundreds or thousands of port and service columns and loses the port-service pairing. The IP-stopping rule preserves contextual relationships among ports and services, and the scoring step aggregates per-scan isolation scores to the IP level, using either the minimum or average score. The paper's implementation adapts the reference Isolation Forest implementation and uses 100 trees for both siForest and the baselines.","core_discovery":"The central claim is that a small change to the Isolation Forest splitting rule fixes a real failure mode for set-structured network data. In ordinary Isolation Forest, trees split until each leaf holds a single scan, so the port-service structure belonging to one IP is scattered across the forest. siForest halts partitioning when every data point in a node belongs to the same IP, making the leaf a set of scans; this keeps the co-occurrence of ports and services for that IP intact while using only a few features. Anomaly scores computed per scan are then combined per IP, with the minimum-score option used in the experiments because it performed better on the generated data. In the reported results, siForest achieves precision 0.285, recall 0.520, and F2-score 0.435 on usage-spike anomalies and precision 0.404, recall 0.709, and F2-score 0.599 on non-standard port-service pairings, making it the only tested method that does not collapse on either anomaly class.","pith_inferences":["The same IP-stopping rule should transfer to other set-structured anomaly detection tasks with a natural group identifier, such as DNS queries per resolver or authentication events per user; the paper does not test these settings.","The paper reports only the minimum-score aggregation, so the average-score variant remains an untested alternative that might trade spike sensitivity for stability in real deployments.","Because summarization dominates on volume anomalies and siForest on configuration anomalies, an ensemble that combines both scores could plausibly raise overall recall beyond either method alone; that combination is not explored.","The generator's tunable frequency and intensity parameters could be used to map detector performance as a function of anomaly strength, giving operators a calibration curve before deployment."],"forward_implications":["A single siForest model can serve as a default detector for scan data when the anomaly type is unknown, since it keeps respectable recall on both volume and configuration anomalies.","IP-level output from siForest means analysts review one score per IP rather than one per scan, aligning the model's output with the attack-surface identification workflow.","Because siForest avoids the thousands of features produced by count-based summarization, it can scale to large-scale scan datasets with modest memory requirements.","The port-service context retained by the IP stopping rule explains why siForest beats summarization on unusual port usage, the anomaly type where counting-based features fail.","The minimum-score aggregation gives siForest sensitivity to single anomalous scans within an IP, which is why its recall on usage-spike anomalies stays high."],"supporting_citations":[{"why":"Defines the original Isolation Forest algorithm whose splitting and anomaly-scoring mechanism siForest modifies.","marker":"[20]"},{"why":"Supplies the reference Isolation Forest implementation that siForest adapts and that provides the baseline comparison.","marker":"[24]"},{"why":"Documents the structure of internet-wide scan data and the common service-port combinations used to generate normal synthetic patterns.","marker":"[6]"},{"why":"Motivates anomaly type 1 as usage spikes associated with scanning or denial-of-service behavior.","marker":"[7]"},{"why":"Motivates anomaly type 2 as non-standard service-port combinations indicating risky or misconfigured services.","marker":"[13]"},{"why":"Provides the feature-engineering principles guiding the flattening and summarization preprocessing baselines.","marker":"[30]"}],"fun_headline_variants":["siForest: set-aware iForest catches both anomaly types","IP-grouped isolation forest only method for both anomaly classes","Preserving IP scan context makes isolation forest catch both anomalies","siForest keeps IP scans intact, alone catching both anomaly classes"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the synthetic data generator's normal service-port combinations and injected anomalies resemble real internet-wide scans closely enough that the ranking of methods transfers to actual network traffic.","fun_headline_variants_meta":{"raw":{"variants":["siForest: set-aware iForest catches both anomaly types","IP-grouped isolation forest only method for both anomaly classes","Preserving IP scan context makes isolation forest catch both anomalies","siForest keeps IP scans intact, alone catching both anomaly classes"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001719,"raw_usage":{"total_tokens":6779,"prompt_tokens":905,"completion_tokens":5874,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":521,"completion_tokens_details":{"reasoning_tokens":5806}},"tokens_in":521,"tokens_out":5874,"duration_ms":39296,"temperature":1.0,"reasoning_tokens":5806,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T20:05:02.235384+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run siForest and the two preprocessing baselines on labeled real internet scan data with known malicious or misconfigured IPs and compare per-IP recall within each anomaly class; if either baseline matches or beats siForest's recall on both usage spikes and non-standard port-service pairs, the consistency claim is falsified.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the original Isolation Forest algorithm whose splitting and anomaly-scoring mechanism siForest modifies."},{"cited_title":"Pedregosa, G","cited_arxiv_id":null,"evidence_quote":"Supplies the reference Isolation Forest implementation that siForest adapts and that provides the baseline comparison."},{"cited_title":"Censys: Internet-wide scan data, 2024","cited_arxiv_id":null,"evidence_quote":"Documents the structure of internet-wide scan data and the common service-port combinations used to generate normal synthetic patterns."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Motivates anomaly type 1 as usage spikes associated with scanning or denial-of-service behavior."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Motivates anomaly type 2 as non-standard service-port combinations indicating risky or misconfigured services."},{"cited_title":"Zheng and A","cited_arxiv_id":null,"evidence_quote":"Provides the feature-engineering principles guiding the flattening and summarization preprocessing baselines."}],"review_version":1}