{"id":"0ef13303-84ef-4a3c-bf7b-7e46bf9ea6f8","arxiv_id":"2502.07119","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"SAFE combines PCA feature ranking, DeepInsight tabular-to-image conversion, masked autoencoder pretraining, and Local Outlier Factor to detect network intrusions without attack labels.","lead":"This paper introduces SAFE, a pipeline that turns network traffic tables into small images, trains a masked autoencoder on normal traffic only, and uses a local outlier detector to flag attacks. The authors report F1-score improvements over two recent intrusion detection systems on four IoT network datasets.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The DeepInsight/t-SNE layout assumption is unverified: with 31 features in a sparse 8x8 grid and stochastic t-SNE, a random-permutation control is needed to show the image mapping contributes to SAFE's reported gains.","rationale":"The reader's weakest assumption is also the most load-bearing one. Module 2 is the only component that distinguishes SAFE from a generic autoencoder-plus-LOF pipeline. The paper's own framing says the goal is 'a spatial relationship between each element' (Proposed Framework), and Figure 4 is offered as evidence, but visual inspection is not a quantitative check. The mathematical setting reinforces the concern: t-SNE minimizes a non-convex KL divergence on only 31 feature-points, so the coordinates are not stable and no seed is reported; the 8x8 matrix has 64 cells, so 33 are empty, and a 75% mask makes many reconstruction targets zero, weakening the supposed spatial signal. The ablation studies in Tables 4 and 5 isolate feature selection and the novelty detector, but not the image-mapping step; no comparison to DeepInsight-CNN or the masked-context-reconstruction IDS leaves the contribution of the layout unmeasured. The proposed permutation control is decisive: if a fixed random layout performs equally, then the apparent success cannot be credited to spatial structure, and the 'image-like format' rationale collapses even if the empirical F1 numbers survive. Since this is exactly the reader's concern and the required experiment is feasible, I keep the CONDITIONAL verdict and set adjustment to UNCHANGED.","tokens_in":11153,"tokens_out":7237,"duration_ms":67085,"concrete_test":"Run the full SAFE pipeline on all four datasets with the DeepInsight/t-SNE pixel placement replaced by a single fixed random permutation of the 64 cells (same seed; k=31, 8x8, masking ratio, MAE architecture, and LOF tuning unchanged). Compare the resulting F1-scores to Table 3. If the permuted-layout F1 is within roughly 1 point on every dataset, the spatial layout is not load-bearing and the paper's explanation needs revision; if it is substantially lower, the layout matters. Also report three t-SNE seeds for the original layout to quantify placement variance.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Module 2 (Vector to Image Matrix Mapping) rests on the assertion that t-SNE/DeepInsight places semantically related features into neighboring 8x8 cells so the MAE's convolutional encoder can exploit spatial structure. This premise is load-bearing: if it fails, SAFE is just an autoencoder over an arbitrary fixed arrangement of 31 features, and the reported F1 gains cannot be attributed to the image conversion. The evidence for the premise is weak. t-SNE is stochastic and non-convex; the coordinates of the 31 feature-points are not a deterministic function of feature similarity, and no seeds or repeated conversions are reported. The chosen 8x8 grid has 64 cells for only 31 selected features, so 33 cells are forced to zero; with a 75% mask, roughly half of the masked pixels are trivially reconstructible zeros, which dilutes the spatial-reasoning signal the MAE is supposed to learn. The paper fixes k=31 and 8x8 for all four datasets ('Experimental Setup, Image Conversion') with no sensitivity analysis, and it does not compare against DeepInsight-CNN (Tran et al. 2021) or the masked-context-reconstruction IDS (Wang et al. 2023), which would isolate the contribution of the image layout. If a random permutation of the 64 cells gives the same F1-scores, the t-SNE layout is not the mechanism and the central claim is overstated; if scores collapse, the layout is essential and the concern is resolved.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SAFE, a self-supervised anomaly detection framework for network intrusion detection. The pipeline consists of PCA-based feature ranking and selection, a DeepInsight/t-SNE vector-to-image mapping, masked autoencoder (MAE) pre-training on normal flows, and a Local Outlier Factor (LOF) novelty detector on the MAE latent representations. SAFE is evaluated on MQTTset, WUSTL-IIoT, X-IIoTID, and Edge-IIoTset against three classical novelty detectors and six deep anomaly detection baselines, reporting F1-score improvements of up to 26.2% over SLAD and 23.5% over Anomal-E. The paper also includes ablations for feature selection, novelty detector choice, and an inference-time overhead analysis.","tokens_in":11468,"tokens_out":2426,"duration_ms":25381,"significance":"If the reported results are reproducible, SAFE would be a practically attractive zero-day intrusion detection approach because it trains only on normal data, uses a lightweight novelty detector, and reports very low per-sample inference overhead. The paper has several strengths: it evaluates on four recent IoT/IIoT datasets, includes a meaningful set of baselines, provides ablations for feature selection and detector choice, and reports a concrete overhead comparison. However, the central empirical claim rests on single-run F1 numbers, the code is not released, the most directly related baselines are omitted, and the load-bearing assumption that the t-SNE/DeepInsight image layout provides useful spatial structure is not tested. These issues are fixable with additional experiments and artifact release, so the contribution is potentially sound but not yet established at the level claimed.","major_comments":[{"comment":"Table 3 reports F1-scores for all methods as single point estimates with no error bars, no number of trials, and no significance tests. Several claimed improvements are tiny in absolute terms (e.g., Edge-IIoTset +0.01% over PCA), and without variance information or statistical testing these differences cannot be distinguished from noise. The authors should run multiple seeds/repetitions, report mean and standard deviation, and perform pairwise significance tests (or at least confidence intervals) for the headline comparisons against SLAD and Anomal-E.","section":"Experiments, Table 3"},{"comment":"The load-bearing premise of Modules 2-3 is that the t-SNE-based DeepInsight layout places semantically related features in neighboring 8x8 cells, so the MAE's convolutional encoder can exploit spatial structure. This premise is not verified. The paper fixes k=31 and an 8x8 grid for all datasets, uses 33 zero-filled cells, and does not report t-SNE seeds or repeated conversions. A random-permutation control (randomly assigning the 31 selected features to grid cells) would directly test whether the spatial layout contributes anything beyond an arbitrary fixed arrangement of features. The authors should also vary k, grid size, mask ratio, and t-SNE initialization to show that the reported performance is not an artifact of these fixed choices.","section":"Experimental Setup, Image Conversion"},{"comment":"The two most closely related methods are not included in the comparison: DeepInsight-CNN (Tran et al., 2021), which also converts tabular network data to images using DeepInsight and classifies with a CNN, and Wang et al.'s masked context reconstruction IDS (Wang et al., 2023), which also uses self-supervised masked reconstruction on network data. Without these baselines, the experiments cannot isolate whether SAFE's gains come from the image conversion, the MAE pre-training, the LOF detector, or the combination. Adding these comparisons, even with numbers taken from the original papers where possible, is necessary to support the claim that SAFE improves on the state of the art in SSL-based intrusion detection.","section":"Baselines, Related Works"},{"comment":"Table 5 selects LOF as the final novelty detector based on F1-score computed on MQTTset, and this same detector choice is then applied to all four datasets. If the MQTTset numbers are computed on the same test set used for the headline comparison, this constitutes test-set-based model selection and can inflate the reported gains. The detector choice should be validated on a held-out validation split, or the authors should report the detector selection procedure as part of a nested optimization and show that the ranking is stable across datasets. A related concern is that Optuna hyperparameter tuning for LOF is mentioned but no details are given about which data split was used for tuning, which data was used for the reported F1, and how many optimization trials were run.","section":"Ablation Study, Table 5"},{"comment":"The paper states a 60/20/20 train/validation/test split but does not describe how samples are subsampled from datasets with millions of rows, whether the split is stratified by class, or whether the training set is restricted to normal samples as implied by Module 1. The reported precision, recall, and F1 values are therefore hard to interpret without knowing the class distribution in each split and whether duplicate flows were removed. The authors should specify the exact preprocessing, sampling, and split procedure for each dataset, including the number of normal and attack samples in train, validation, and test sets.","section":"Experimental Setup, Data Division"}],"minor_comments":[{"comment":"The header 'Y ear' should read 'Year'.","section":"Table 2"},{"comment":"The notation R(x_i^train) is defined with an index i that overloads the per-sample index; the definition is understandable but should use a distinct feature index for clarity.","section":"Proposed Framework, Module 1"},{"comment":"Figure 5 combines four datasets and many method labels in two panels, making it difficult to map which marker corresponds to which method and dataset; the caption should explain the grouping and marker conventions, and the panels would be clearer if plotted per dataset.","section":"Results, Figure 5"},{"comment":"References [23] and [24] in the arXiv version both point to SLAD with slightly different titles; the duplicate should be merged into a single reference.","section":"References"},{"comment":"The conclusion restates improvements of 26.15% and 23.52% over SLAD and Anomal-E, while the abstract and Table 3 are based on dataset-specific maxima and averages; the text should consistently distinguish 'up to', 'on average', and the specific dataset from which each number comes.","section":"Conclusion"},{"comment":"No code or repository is provided, and the paper does not state whether artifacts will be released; given that the main contribution is an empirically evaluated pipeline, releasing code and processed datasets would substantially increase reproducibility.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper is likely to be of interest to the IDS and self-supervised learning community, but the empirical claims are currently under-supported by the experimental methodology. The missing comparisons to DeepInsight-CNN and Wang et al.'s masked reconstruction model are particularly important because they directly test the novelty of the proposed components. I would also flag that the novelty-detector ablation on MQTTset appears to be a test-set-based model selection step; this should be clarified or corrected before publication. These issues are addressable within the scope of a revision, so I recommend major_revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take on arXiv:2502.07119. The paper builds a four-stage pipeline — PCA feature ranking, DeepInsight/t-SNE image conversion, masked autoencoder, LOF — and evaluates it on four IoT IDS datasets. As a package it's new, and the writing is unusually clear. The ablations they do include (feature selection on/off, detector choice) are genuinely useful. They also report inference time relative to flow duration, which is the right way to frame deployment overhead. Credit where due: the engineering is sensible and the evaluation covers a reasonable set of standard baselines.\n\nThe soft spots are real and concentrated in the evaluation. The central claim — 'outperforms SLAD by up to 26.2% and Anomal-E by up to 23.5% F1' — rests on single-run numbers in Table 3 with no error bars, no significance tests, and no code. The largest gains are on MQTTset, but that's also the dataset where they did model selection for the LOF detector (Table 5). That's textbook selection-on-the-test-set, even if unintentional. On Edge-IIoTset the gain over the next best is 0.01%, which the paper honestly reports, but it undermines the 'consistently outperforms' phrasing.\n\nThe deeper issue, which I share with the stress-test note, is that the one genuinely novel module — the t-SNE image mapping — is never isolated. They don't compare against DeepInsight-CNN (Tran et al. 2021) or against Wang et al.'s masked-context-reconstruction IDS, which are the two closest priors. Without a control that permutes the 8x8 grid or replaces t-SNE with a fixed random layout, you cannot tell whether the MAE's convolutional encoder is exploiting genuine spatial structure or just fitting a fixed arrangement of 31 features. The fact that 33 of 64 cells are always zero, and a 75% mask makes about half the masked pixels trivially reconstructible, only strengthens the need for that control.\n\nNone of this is fatal. The pipeline is plausible, and the missing experiments are all doable. A serious referee could ask for seeds, error bars, the two missing baselines, and a permutation test. I'd send it to review rather than desk-reject. For my own work, I wouldn't cite it until code and the controls appear, but I'd keep it on the reading list as a good example of how to frame self-supervised IDS evaluation.","headline":"A cleanly-written pipeline paper whose headline F1 gains are plausible but not yet substantiated: no code, no error bars, and no control that isolates the t-SNE image layout, which is the one genuinely novel piece.","tokens_in":12064,"tokens_out":2228,"would_cite":false,"duration_ms":19281,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Self-supervised pipeline that turns network flows into images beats top anomaly detectors by up to 26% in F1.","keywords":["self-supervised learning","anomaly detection","intrusion detection","masked autoencoder","Internet of Things","Local Outlier Factor","tabular-to-image conversion","zero-day attack detection"],"falsifier":"Take SAFE and shuffle the pixel assignments in the 8-by-8 grid while keeping all other hyperparameters fixed, then retrain and measure F1 on the same four datasets; if F1 does not drop substantially, the spatial layout carries no information and the image-conversion step is not the cause of the reported gains. A complementary check is to run the same MAE+LOF pipeline on the top-31 features in their original vector form: matching performance would show the image transform is unnecessary.","tokens_in":10905,"feed_emoji":"🛡️","tokens_out":12236,"duration_ms":88077,"temperature":0.7,"pith_summary":"This paper introduces SAFE, a self-supervised pipeline that learns to detect network intrusions from normal traffic alone, without attack labels. Its central move is to turn tabular flow features into compact 8-by-8 image-like matrices, letting a masked autoencoder learn the spatial texture of normal behavior; a Local Outlier Factor then scores new flows against that learned representation. On four IoT and industrial-IoT benchmarks, SAFE reports F1-scores from 91.38% to 99.95% and claims it beats SLAD by up to 26.2% and Anomal-E by up to 23.5% in F1. If right, this is a label-free route to zero-day attack detection with negligible per-flow overhead: about 0.18 ms per sample on X-IIoTID, roughly 3.65% of the median flow duration.","feed_headline":"Turning network flows into images boosts attack F1 by 26%","feed_subtitle":"SAFE trains a masked autoencoder on normal traffic only, then flags unseen attacks as outliers — no labels needed.","key_machinery":"The load-bearing mechanism is the vector-to-image mapping. PCA computes each feature's importance as the summed absolute loadings across principal components that explain 95% of the variance, retaining the top $k=31$ features; t-SNE places those features in the plane, and DeepInsight's convex-hull binning assigns each feature to a unique pixel in an $8 \\times 8$ grayscale matrix. This construction lets a masked autoencoder treat a network flow as an image and learn spatial correlations between neighboring features. The MAE encoder's latent vector is the representation that LOF scores, and random 75% masking with mean-squared-error reconstruction supplies the label-free training signal.","core_discovery":"SAFE's core claim is that a masked autoencoder trained on image-like encodings of normal network flows produces a latent space in which attacks are outliers, even when those attacks were never seen during training. The paper supports this by ranking features with PCA loadings, embedding the top 31 features into an 8-by-8 grid via DeepInsight's t-SNE-based layout, and training the MAE to reconstruct 75%-masked images. The encoder's bottleneck vectors are then fed to LOF, which separates normal from anomalous flows. Across MQTTset, WUSTL-IIoT, X-IIoTID, and Edge-IIoTset, this combination outperforms nine baselines, including graph-based Anomal-E and scale-learning SLAD, and the authors attribute the gain to the layered pipeline of feature selection, spatial mapping, and representation learning.","pith_inferences":["A test the paper does not run is a sensitivity analysis on the image geometry: if randomly permuting the 8x8 pixel positions preserves F1, then spatial arrangement is not the source of the gain; if it collapses, the t-SNE layout is doing real work.","The same pipeline could be tried on other tabular anomaly domains, such as financial fraud or system logs, whenever feature correlations can be laid out spatially; the paper only demonstrates network traffic.","The ablation shows feature selection adds only 0.13-1.49% F1, so the bulk of the improvement over baselines likely comes from the image conversion plus MAE rather than PCA ranking; a raw-feature MAE baseline would isolate each contribution.","Because DeepInsight is fitted on normal training data, the pixel layout is dataset-specific; deployment would need a policy for updating the layout under concept drift, which the paper does not address."],"forward_implications":["Zero-day detection becomes label-free: only normal flows are needed for training, and attack types never seen in training still separate from normal flows in the learned latent space.","The same fixed configuration (top-31 features, 8x8 grid, 75% masking) transfers across four heterogeneous IoT/IIoT datasets with F1-scores above 91%, so the pipeline is not tied to a single protocol or attack family.","Per-sample inference cost (0.1819 ms on X-IIoTID) is small relative to the median flow duration of 4.98 ms, supporting real-time deployment.","The novelty detector matters but is not the whole story: swapping LOF for IF, PCA, SLAD, ICL, or RDP on MQTTset lowers F1 by 1.3 to 11.7 percentage points, so the learned representation helps multiple downstream detectors.","Consistently top-two precision and top-three recall across datasets indicates the method balances false positives and false negatives, which matters when missing an attack is costlier than a false alarm."],"supporting_citations":[{"why":"Supplies the DeepInsight method that maps tabular feature vectors to image matrices.","marker":"Tran et al. 2021"},{"why":"Provides the t-SNE embedding that DeepInsight uses to place features in 2D.","marker":"Van der Maaten and Hinton 2008"},{"why":"Defines the masked autoencoder architecture and pretraining scheme SAFE adapts.","marker":"He et al. 2022"},{"why":"Supplies the Local Outlier Factor detector used in the final novelty-detection module.","marker":"Cheng, Zou, and Dong 2019"},{"why":"Is the graph-based self-supervised intrusion detection baseline whose F1 SAFE claims to beat by up to 23.5%.","marker":"Caville et al. 2022"},{"why":"Is the scale-learning anomaly detection baseline whose F1 SAFE claims to beat by up to 26.2%.","marker":"Xu et al. 2023b"},{"why":"Provides MQTTset, one of the four evaluation datasets.","marker":"Vaccari et al. 2020"},{"why":"Provides WUSTL-IIoT, one of the four evaluation datasets.","marker":"Zolanvari et al. 2021"},{"why":"Provides X-IIoTID, one of the four evaluation datasets.","marker":"Al-Hawawreh, Sitnikova, and Aboutorab 2022"},{"why":"Provides Edge-IIoTset, one of the four evaluation datasets.","marker":"Ferrag et al. 2022"}],"fun_headline_variants":["Self-supervised SAFE beats state-of-the-art IDS by 26%","Masked autoencoders on network images spike F1 scores","SAFE: no labels needed, beats top IDS by up to 26%","SAFE self-supervised framework ups intrusion F1 by 26%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that placing the selected features on an 8-by-8 grid puts related features next to each other, so the masked autoencoder learns something more useful than it would from the raw feature list; if the pixel layout is arbitrary or dataset-specific, the image conversion adds little and the reported gains would not transfer.","fun_headline_variants_meta":{"raw":{"variants":["Self-supervised SAFE beats state-of-the-art IDS by 26%","Masked autoencoders on network images spike F1 scores","SAFE: no labels needed, beats top IDS by up to 26%","SAFE self-supervised framework ups intrusion F1 by 26%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000742,"raw_usage":{"total_tokens":3287,"prompt_tokens":899,"completion_tokens":2388,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":515,"completion_tokens_details":{"reasoning_tokens":2306}},"tokens_in":515,"tokens_out":2388,"duration_ms":14429,"temperature":1.0,"reasoning_tokens":2306,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T13:45:48.229639+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take SAFE and shuffle the pixel assignments in the 8-by-8 grid while keeping all other hyperparameters fixed, then retrain and measure F1 on the same four datasets; if F1 does not drop substantially, the spatial layout carries no information and the image-conversion step is not the cause of the reported gains. A complementary check is to run the same MAE+LOF pipeline on the top-31 features in their original vector form: matching performance would show the image transform is unnecessary.","supporting_citations":[{"cited_title":"A.; et al","cited_arxiv_id":null,"evidence_quote":"Provides Edge-IIoTset, one of the four evaluation datasets."},{"cited_title":"P.; Nguyen, V","cited_arxiv_id":null,"evidence_quote":"Supplies the DeepInsight method that maps tabular feature vectors to image matrices."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Local Outlier Factor detector used in the final novelty-detection module."},{"cited_title":"W.; Layeghy, S.; and Portmann, M","cited_arxiv_id":null,"evidence_quote":"Is the graph-based self-supervised intrusion detection baseline whose F1 SAFE claims to beat by up to 23.5%."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides MQTTset, one of the four evaluation datasets."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides WUSTL-IIoT, one of the four evaluation datasets."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides X-IIoTID, one of the four evaluation datasets."}],"review_version":1}