siForest: Detecting Network Anomalies with Set-Structured Isolation Forest

Christie Djidjev

REVIEW 4 major objections 5 minor 32 references

siForest: Detecting Network Anomalies with Set-Structured Isolation Forest

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read siForest, an Isolation Forest variant that stops splitting at IP boundaries, is the only tested method that detects both volume spikes and unusual port-service pairings consistently.

desk verdict A simple, clearly-described variant of iForest for IP-level scan data; the consistency claim is plausible but the reported evidence is too thin to support it. read the letter →

arxiv 2412.06015 v1 pith:K26RRB6Y submitted 2024-12-08 cs.LG cs.CR

Christie Djidjev This is my paper · ORCID

classification cs.LGcs.CR

keywords siForestIsolationset-structureddatanetworkanomalydetectioninternetscanservice-portanomaliesunsupervisedattacksurfaceidentification

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that internet scan data is best treated as sets, with each IP holding its own list of ports and services, rather than as flattened rows or per-IP count vectors. It introduces siForest, a version of Isolation Forest whose trees stop splitting once a node contains scans from only one IP, then aggregate per-scan anomaly scores up to the IP level. On synthetic data modeled after internet-wide scans, siForest is the only one of the three tested methods that keeps reasonably strong precision, recall, and F2-score on both anomaly types: usage spikes and non-standard service-port combinations. If correct, this gives network defenders a low-dimensional, IP-level detector that does not require knowing the anomaly type in advance.

What carries the argument

The Set-Partitioned Isolation Forest (siForest) is an Isolation Forest whose tree construction stops splitting when all data points in a node belong to the same IP address, so each leaf is a set of scans from one IP. This contrasts with two preprocessing baselines: flattening, which expands each scan into IP-port-service rows and thus loses the IP grouping, and summarization, which creates one row per IP with counts for hundreds or thousands of port and service columns and loses the port-service pairing. The IP-stopping rule preserves contextual relationships among ports and services, and the scoring step aggregates per-scan isolation scores to the IP level, using either the minimum or average score. The paper's implementation adapts the reference Isolation Forest implementation and uses 100 trees for both siForest and the baselines.

What would settle it

Run siForest and the two preprocessing baselines on labeled real internet scan data with known malicious or misconfigured IPs and compare per-IP recall within each anomaly class; if either baseline matches or beats siForest's recall on both usage spikes and non-standard port-service pairs, the consistency claim is falsified.

Watch

Extended reading notes

Core claim

The central claim is that a small change to the Isolation Forest splitting rule fixes a real failure mode for set-structured network data. In ordinary Isolation Forest, trees split until each leaf holds a single scan, so the port-service structure belonging to one IP is scattered across the forest. siForest halts partitioning when every data point in a node belongs to the same IP, making the leaf a set of scans; this keeps the co-occurrence of ports and services for that IP intact while using only a few features. Anomaly scores computed per scan are then combined per IP, with the minimum-score option used in the experiments because it performed better on the generated data. In the reported results, siForest achieves precision 0.285, recall 0.520, and F2-score 0.435 on usage-spike anomalies and precision 0.404, recall 0.709, and F2-score 0.599 on non-standard port-service pairings, making it the only tested method that does not collapse on either anomaly class.

Load-bearing premise

The load-bearing premise is that the synthetic data generator's normal service-port combinations and injected anomalies resemble real internet-wide scans closely enough that the ranking of methods transfers to actual network traffic.

Editorial extensions

If this is right

A single siForest model can serve as a default detector for scan data when the anomaly type is unknown, since it keeps respectable recall on both volume and configuration anomalies.
IP-level output from siForest means analysts review one score per IP rather than one per scan, aligning the model's output with the attack-surface identification workflow.
Because siForest avoids the thousands of features produced by count-based summarization, it can scale to large-scale scan datasets with modest memory requirements.
The port-service context retained by the IP stopping rule explains why siForest beats summarization on unusual port usage, the anomaly type where counting-based features fail.
The minimum-score aggregation gives siForest sensitivity to single anomalous scans within an IP, which is why its recall on usage-spike anomalies stays high.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The same IP-stopping rule should transfer to other set-structured anomaly detection tasks with a natural group identifier, such as DNS queries per resolver or authentication events per user; the paper does not test these settings.
The paper reports only the minimum-score aggregation, so the average-score variant remains an untested alternative that might trade spike sensitivity for stability in real deployments.
Because summarization dominates on volume anomalies and siForest on configuration anomalies, an ensemble that combines both scores could plausibly raise overall recall beyond either method alone; that combination is not explored.
The generator's tunable frequency and intensity parameters could be used to map detector performance as a function of anomaly strength, giving operators a calibration curve before deployment.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The core idea in this paper is modest but clean: modify Isolation Forest so that it stops splitting when all the scans in a node belong to the same IP, then aggregate the resulting leaf scores per IP. That gives you an IP-level anomaly detector without flattening the scan lists or exploding the feature space with one-hot summaries. I think that is genuinely new and it is described clearly enough to reimplement. The two anomaly types—volume spikes and non-standard service-port pairs—are reasonable, and the paper is honest that recall matters more than precision for this use case.

The problem is the evidence for the headline claim. The paper says siForest is the only method with consistently good performance across both anomaly types. What we actually get is a single precision/recall/F2 point per method per anomaly type, with no standard deviation, no interval, no per-seed breakdown, and no significance test, even though the setup says 10 random datasets were generated for each type. On top of that, Section 4.1 admits the min-score aggregation rule was chosen because it gives better performance on the generated data. That is post hoc selection on the evaluation set, which biases the comparison in siForest's favor. And the numbers are not that convincing: for type 1, siForest has F2 0.435 versus summarization's 0.59, and its precision is 0.285. So "consistent" is doing a lot of work; it means siForest is mediocre at both types while each baseline is good at one type and bad at the other. That may actually be a useful practical property, but it is not the same as being consistently good, and without variance estimates we cannot even tell whether the relative ordering is stable across seeds.

I agree with the stress-test note that this is the load-bearing weakness. The fix is straightforward: report per-seed results with error bars, pre-specify the aggregation rule or use a validation split to select it, and try at least one real scan dataset, even a small sample. The synthetic generator's resemblance to Censys is asserted but never checked, so external validity is unknown.

There are some minor issues as well: a typo in "Experiemntal," a related-work section that reads more like a citation dump than a positioning, and the figures are basic. None of these are fatal.

Who is this for? Someone working on network anomaly detection who wants a cheap IP-level baseline to compare against. I would not cite it for the empirical results, but I would cite it for the algorithm idea if I needed a set-structured iForest variant. The paper deserves a serious referee because the idea is simple, plausible, and testable; peer review could push it into a decent short paper. As is, I would not accept it without major revision.

Referee Report

4 major / 5 minor

Summary. The paper proposes siForest, a modification of the Isolation Forest algorithm for set-structured network scan data. The key idea is to stop tree partitioning when all data points in a node belong to the same IP address, thereby preserving the contextual relationship between ports and services while operating at the IP level. Two IP-level scoring aggregation strategies (minimum and average) are introduced. The algorithm is compared against two iForest baselines using different preprocessing approaches (flattening and summarization) on synthetic datasets that mimic Censys internet scans, with two anomaly types: usage spikes (type 1) and non-standard service-port combinations (type 2). The paper reports precision, recall, and F2-score for each method and anomaly type, and concludes that siForest is the only method showing consistently good performance across both anomaly types.

Significance. If the central claim were fully supported, siForest would be a practically useful extension of Isolation Forest for IP-level anomaly detection on network scan data, because it retains the low feature dimensionality of the flattened representation while producing IP-level predictions. The algorithmic modification is simple, clearly described, and plausible, and the comparison against two natural preprocessing baselines on identically generated data is a reasonable starting point. However, the significance is currently limited by the evaluation: the experiments are entirely synthetic, no variance or significance measures are reported, the best-performing aggregation rule appears to have been selected post hoc on the same data, and the reported type-1 precision for siForest (0.285) is low. These issues mean that the paper's main claim is not yet established, although the work is a promising direction that could be strengthened with additional experiments and statistical reporting.

major comments (4)

[Section 4.1 and Section 4.4] The paper states in Section 4.1 that 10 different random datasets are generated for each anomaly type, but Section 4.4 reports only single precision, recall, and F2 values for each method, with no standard deviations, confidence intervals, per-seed results, or significance tests. Without this information, the claim that siForest is 'the only methods among the tested ones that shows consistently good performance' (Section 5) is not statistically supported. The authors should report per-dataset results and either paired statistical tests or at least variance measures across the 10 runs, so that the reader can assess whether the observed differences are stable.
[Section 4.1] The authors write: 'We calculate anomaly scores at the IP level using the minimum score option as it gives a better performance for the generated data.' This sentence indicates that the aggregation rule was selected based on its performance on the same generated data used for evaluation. With only two anomaly types, such post hoc selection can inflate the apparent advantage of siForest, since the reported metrics are the result of choosing the more favorable rule. To make the comparison fair, the authors should either justify the minimum aggregation rule on independent grounds, or report results for both the minimum and average aggregation strategies for both anomaly types, and discuss sensitivity.
[Section 4.4, Figure 6, and Section 5] The claim that siForest 'comes close' to the summarization approach for type 1 anomalies is not well supported by the reported numbers. For type 1, siForest has precision 0.285, recall 0.520, and F2 0.435, while summarization has precision 0.61, recall 0.58, and F2 0.59. The F2 difference (0.435 vs. 0.59) is substantial, and the precision is less than half. The conclusion that siForest is 'consistently good' depends on a threshold of what counts as acceptable performance; the paper should either justify such a threshold, report a combined metric that accounts for precision and recall more transparently, or soften the conclusion to reflect that siForest is second-best on type 1 anomalies.
[Section 4.2 and Section 5] The external validity of the evaluation is limited by the use of synthetic data. Section 4.2 states that normal network behavior is based on 'common service-port combinations documented in Censys,' but no validation is provided that the generated data distribution resembles real Censys scans in terms of frequencies, correlations, or anomaly characteristics. Section 5 acknowledges that testing on real-world datasets is future work, but this limitation is load-bearing for the paper's central claim: if the synthetic distribution differs from real network traffic, the performance rankings may not transfer. The authors should either validate the synthetic data against real scans (for example, by comparing marginal distributions or using a small labeled real dataset) or explicitly restrict the conclusion to the synthetic setting.

minor comments (5)

[Section 4.1] The section heading contains a typo: 'Experiemntal Setup' should be 'Experimental Setup.'
[Section 2] The sentence 'It’s main advantages are the high computational efficiency' should use 'Its main advantages' instead of 'It’s main advantages.'
[Section 5] The sentence 'siForest is the only methods among the tested ones' has a subject-verb agreement error and should read 'the only method.'
[Abstract and Section 5] The abstract says siForest has 'the potential to outperform traditional approaches on some types of internet scan data,' while Section 5 makes the stronger claim that it is the only method with consistently good performance. The strength of the language should be aligned with the actual evidence.
[Figure 6] The two plots in Figure 6 are not accompanied by a description of their axes, legends, or error bars. Adding axis labels, method labels, and variance information would make the figure self-contained and easier to interpret.

Circularity Check

1 steps flagged · score 2.0 of 10

No derivation-level circularity; the only closed loop is the disclosed post hoc selection of the minimum-score aggregation rule, which mildly inflates the headline comparison.

fitted input called prediction [Section 4.1 (Experimental Setup); Section 5 (Conclusions and Future Work)]
"We calculate anomaly scores at the IP level using the minimum score option as it gives a better performance for the generated data. ... siForest is the only methods among the tested ones that shows consistently good performance over all anomaly types."

The IP-level score aggregation rule (minimum vs. average) is a component of siForest, and Section 4.1 discloses that it was chosen because it gave better performance on the generated data. Section 5 then uses a performance comparison on that same generated data to support the claim that siForest is the only method with consistently good performance. Thus the evaluation metric was used to select the rule whose outcome is later reported as evidence; the comparison does not independently test siForest against the data used for that choice. Without per-seed breakdowns, confidence intervals, or an average-score ablation, the reported advantage may be partly an artifact of this closed selection loop.

full rationale

The paper makes no first-principles derivation claim; siForest is an algorithmic extension of standard Isolation Forest, and its tree construction and scoring follow the established iForest framework. There are no load-bearing self-citations, no imported uniqueness theorems, and no ansatz smuggled in via citation. The only circularity-adjacent element is the disclosed choice of the minimum-score aggregation rule because it performed better on the generated data, followed by a headline performance claim evaluated on that same data. This is a genuine limitation of the empirical comparison, but it does not reduce the algorithm's definition or the anomaly scores to the reported outcome by construction; the central comparison retains independent algorithmic content. The synthetic-data realism and lack of statistical error bars are external-validity and reporting concerns rather than circularity, so the score is modest.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central claim rests on the representativeness of the synthetic data and on the validity of the modified scoring. No numeric constants are fitted to real data, but the aggregation rule is selected post hoc from test performance, adding a mild free-parameter burden.

free parameters (2)

anomaly_score_aggregation = min
The paper uses the minimum anomaly score among scans per IP, chosen because it "gives a better performance for the generated data" (Section 4.1). This is a post hoc selection rather than a principled choice.
number_of_trees = 100
Set to 100 for consistency across methods but not justified as optimal (Section 4.1).

assumptions (3)

domain assumption Synthetic data generated from common Censys service-port combinations is representative of real network scan data.
Section 4.2 states normal behavior is based on documented Censys pairings, but no real data is used; results may not generalize.
domain assumption Isolation Forest path-length scoring remains a valid anomaly signal when splitting is stopped at IP-pure leaves.
Section 3.3 modifies the stopping rule but provides no theoretical argument that the resulting path lengths preserve anomaly separation.
domain assumption The IP-level aggregation via the minimum score preserves IP-level anomalousness.
Section 3.3 introduces min and average strategies; the paper assumes min is meaningful, but it is chosen by performance.

how reviews work

0 comments

Cite this review

Pith. "Pith review of siForest: Detecting Network Anomalies with Set-Structured Isolation Forest." pith.science (2026). https://pith.science/paper/K26RRB6Y

@misc{pith2026241206015,
  author       = {Pith},
  title        = {Pith review of: siForest: Detecting Network Anomalies with Set-Structured Isolation Forest},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/K26RRB6Y}},
  note         = {Machine review of arXiv:2412.06015}
}

read the original abstract

As cyber threats continue to evolve in sophistication and scale, the ability to detect anomalous network behavior has become critical for maintaining robust cybersecurity defenses. Modern cybersecurity systems face the overwhelming challenge of analyzing billions of daily network interactions to identify potential threats, making efficient and accurate anomaly detection algorithms crucial for network defense. This paper investigates the use of variations of the Isolation Forest (iForest) machine learning algorithm for detecting anomalies in internet scan data. In particular, it presents the Set-Partitioned Isolation Forest (siForest), a novel extension of the iForest method designed to detect anomalies in set-structured data. By treating instances such as sets of multiple network scans with the same IP address as cohesive units, siForest effectively addresses some challenges of analyzing complex, multidimensional datasets. Extensive experiments on synthetic datasets simulating diverse anomaly scenarios in network traffic demonstrate that siForest has the potential to outperform traditional approaches on some types of internet scan data.

Figures

Figures reproduced from arXiv: 2412.06015 by the authors.

**Figure 2.** Comparison of the number of features and rows across three different data represen [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

**Figure 3.** Comparison of Isolation Forest approaches showing datapoint types, features, and [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Figures from the paper (3 more)

**Figure 4.** Figure 4: Bars in red indicate anomalous IP scans with much higher usage compared to [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]

**Figure 5.** Figure 5: Red bars highlight anomalous behavior based on atypical port usage. [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]

**Figure 6.** Figure 6: Comparison of detection performance across anomaly types and methods. [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

32 extracted references · 28 canonical work pages

[1]

C. C. Aggarwal. Outlier Analysis. Springer, 2017

work page 2017
[2]

Ahmed, A

M. Ahmed, A. N. Mahmood, and J. Hu. A survey of network anomaly detection techniques. Journal of Network and Computer Applications, 60:19–31, 2016. 13

work page 2016
[3]

Akoglu, H

L. Akoglu, H. Tong, and D. Koutra. Graph-based anomaly detection and description: A survey. Data Mining and Knowledge Discovery, 29(3):626–688, 2015

work page 2015
[4]

Barnett and T

V. Barnett and T. Lewis. Outliers in statistical data. Journal of Applied Statistics, 4(1):121–129, 1974

work page 1974
[5]

Bengio, A

Y. Bengio, A. Courville, and P. Vincent. Representation learning: A review and new perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(8):1798–1828, 2013

work page 2013
[6]

Censys: Internet-wide scan data, 2024

Censys. Censys: Internet-wide scan data, 2024. https://censys.io

work page 2024
[7]

F. H. Center. Dns destination anomaly detection. https://help.fortinet.com/ fsiem/Public_Resource_Access/7_1_1/rules/PH_RULE_ANOMALY_DNS_DEST.htm,

work page
[8]

Chalapathy and S

R. Chalapathy and S. K. Chawla. Deep learning for anomaly detection: A survey. arXiv preprint arXiv:1901.03407, 2019

arXiv 1901

Show all 32 references

[9]

Chandola, A

V. Chandola, A. Banerjee, and V. Kumar. Anomaly detection: A survey. ACM Computing Surveys, 41(3):1–58, 2009

2009
[10]

Chandola, A

V. Chandola, A. Banerjee, and V. Kumar. Data mining for cybersecurity applications. IEEE Transactions on Knowledge and Data Engineering, 23(3):401–414, 2011

2011
[11]

Domingos

P. Domingos. A few useful things to know about machine learning. Communications of the ACM, 55(10):78–87, 2012

2012
[12]

Ester, H.-P

M. Ester, H.-P. Kriegel, J. Sander, X. Xu, et al. A density-based algorithm for discovering clusters in large spatial databases with noise. In KDD, volume 96, pages 226–231, 1996

1996
[13]

H.-T. Geek. Why are some ports risky, and how do you secure them? https://www.howtogeek.com/devops/ why-are-some-ports-risky-and-how-do-you-secure-them/ , 2023. Accessed: 2024-11-25. 14

2023
[14]

Hariri, M

S. Hariri, M. Kind, and R. J. Brunner. Extended isolation forest. arXiv preprint arXiv:1811.02141, 2019

2019 arXiv
[15]

C. R. Harris, K. J. Millman, S. J. van der Walt, et al. Numpy: The fundamental package for scientific computing with python. https://numpy.org/, 2020

2020
[16]

T. K. Ho. Random decision forests. In Proceedings of 3rd international conference on document analysis and recognition, volume 1, pages 278–282. IEEE, 1995

1995
[17]

Jin and J

X. Jin and J. Han. K-Means Clustering, pages 563–564. Springer US, Boston, MA, 2010

2010
[18]

T. N. Kipf and M. Welling. Semi-supervised classification with graph convolutional networks. In Proceedings of the International Conference on Learning Representations (ICLR), 2016

2016
[19]

LeCun, Y

Y. LeCun, Y. Bengio, and G. Hinton. Deep learning. Nature, 521(7553):436–444, 2015

2015
[20]

F. T. Liu, K. M. Ting, and Z.-H. Zhou. Isolation forest. In Proceedings of the IEEE International Conference on Data Mining (ICDM), pages 413–422, 2008

2008
[21]

F. T. Liu, K. M. Ting, and Z.-H. Zhou. Isolation-based anomaly detection. ACM Transactions on Knowledge Discovery from Data (TKDD), 6(1):3, 2012

2012
[22]

Markou and S

M. Markou and S. Singh. Novelty and anomaly detection: A review. Signal Processing, 83(12):2481–2521, 2003

2003
[23]

McKinney et al

W. McKinney et al. pandas: Python data analysis library. https://pandas.pydata.org/, 2010

2010
[24]

Pedregosa, G

F. Pedregosa, G. Varoquaux, A. Gramfort, et al. Scikit-learn: Machine learning in python. https://scikit-learn.org/, 2011

2011
[25]

Sommer and V

R. Sommer and V. Paxson. Outside the closed world: On using machine learning for network intrusion detection. In Proceedings of the IEEE Symposium on Security and Privacy, pages 305–316, 2010. 15

2010
[26]

J. Xie, R. Girshick, and A. Farhadi. Unsupervised deep embedding for clustering analysis. In Proceedings of the International Conference on Machine Learning (ICML), 2017

2017
[27]

Yu et al

J. Yu et al. Network traffic analysis: Anomaly detection using deep learning techniques. IEEE Transactions on Dependable and Secure Computing, 15(5):842–853, 2018

2018
[28]

Zaheer et al

M. Zaheer et al. Deep sets. In Advances in Neural Information Processing Systems (NeurIPS), pages 3391–3401, 2017

2017
[29]

Zhao et al

L. Zhao et al. Learning graph representations with node features for link prediction. arXiv preprint arXiv:1906.04733, 2019

1906 arXiv
[30]

Zheng and A

A. Zheng and A. Casari. Feature engineering for machine learning: principles and techniques for data scientists. ” O’Reilly Media, Inc.”, 2018

2018
[31]

Zhu et al

W. Zhu et al. Synthetic data generation for machine learning in network security. IEEE Access, 10:101345–101360, 2022. 16

2022
[2023]

Accessed: 2024-11-25

2024

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.

[1] [1]

C. C. Aggarwal. Outlier Analysis. Springer, 2017

work page 2017

[2] [2]

Ahmed, A

M. Ahmed, A. N. Mahmood, and J. Hu. A survey of network anomaly detection techniques. Journal of Network and Computer Applications, 60:19–31, 2016. 13

work page 2016

[3] [3]

Akoglu, H

L. Akoglu, H. Tong, and D. Koutra. Graph-based anomaly detection and description: A survey. Data Mining and Knowledge Discovery, 29(3):626–688, 2015

work page 2015

[4] [4]

Barnett and T

V. Barnett and T. Lewis. Outliers in statistical data. Journal of Applied Statistics, 4(1):121–129, 1974

work page 1974

[5] [5]

Bengio, A

Y. Bengio, A. Courville, and P. Vincent. Representation learning: A review and new perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(8):1798–1828, 2013

work page 2013

[6] [6]

Censys: Internet-wide scan data, 2024

Censys. Censys: Internet-wide scan data, 2024. https://censys.io

work page 2024

[7] [7]

F. H. Center. Dns destination anomaly detection. https://help.fortinet.com/ fsiem/Public_Resource_Access/7_1_1/rules/PH_RULE_ANOMALY_DNS_DEST.htm,

work page

[8] [8]

Chalapathy and S

R. Chalapathy and S. K. Chawla. Deep learning for anomaly detection: A survey. arXiv preprint arXiv:1901.03407, 2019

arXiv 1901