{"id":"75a925a2-e7ff-4ebf-ac97-761652bb7a99","arxiv_id":"2501.15266","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":2.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"A decision tree trained on six autoencoder-learned features reports 99.94% accuracy and F1 on the Edge-IIoTset benchmark, with 0.185 ms per-sample inference on a Jetson Nano.","lead":"The authors built an autoencoder to compress network-traffic features and then trained six classifiers on the compressed data; a decision tree reached 99.94% accuracy on the Edge-IIoTset benchmark. They also measured inference latency on a Jetson Nano edge device at about 0.185 ms per sample for binary classification and 0.187 ms for multiclass classification.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported 99.94% Decision Tree accuracy is internally inconsistent with XGB/LightGBM on the same autoencoder features; boosted trees should match a single tree if the features really separate the 15 classes, so the gap points to leakage/evaluation error rather than a validated method.","rationale":"I read the paper as claiming that a lightweight, cost-sensitive autoencoder produces six features on which a decision tree attains 99.94% F1 on Edge-IIoTset, enabling edge deployment. For this to be true, the six features must be almost perfectly discriminative across 15 imbalanced classes, and the evaluation must be leak-free. The first condition is directly in tension with Table 4: XGBoost and LightGBM, which should be able to match or exceed a single tree on the same tabular features, obtain only 81.81% and 79.46%. That internal inconsistency is more diagnostic than any external prior about achievable accuracy. It indicates the DT result is not produced by a general feature representation; otherwise boosted ensembles would extract the same separation. The paper's missing split definition, absent ablation, and lack of code mean this cannot be checked post hoc, so the strongest claim is not credible on the evidence. This partially agrees with the reader's weakest assumption: leakage is a plausible mechanism, but the decisive evidence is the DT-versus-boosted-tree gap, not the preprocessing ambiguity alone. Therefore I would leave the reader's reject verdict unchanged.","tokens_in":8409,"tokens_out":4148,"duration_ms":38280,"concrete_test":"Re-run the pipeline with a fixed random split and a single code path: fit the correlation filter and autoencoder on the training split only, then train DT, XGB, and LGBM on the same six features with a modest hyperparameter search (e.g., XGB with n_estimators=300, max_depth=6, learning_rate=0.1). If XGB/LGBM remain near 0.80 while DT is 0.9994, inspect the DT branch for leakage (e.g., test labels used in feature selection, class-weight computation over the full dataset, or accidental target encoding). Require per-class F1 for rare classes (MITM, Fingerprinting) and a completed confusion matrix; this settles whether the headline is reproducible or an artifact.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that the six autoencoder features are so informative that a single decision tree reaches 99.94% accuracy and F1 across 15 highly imbalanced classes on Edge-IIoTset. The same feature set is then given to XGBoost and LightGBM, which obtain only 81.81% and 79.46% in Table 4. This 18-20 point gap is the load-bearing problem: for tabular data, gradient-boosted trees should be able to match or exceed a single tree, and with even default hyperparameters they should not be twenty points worse if the features are nearly perfectly class-separable. A gap of this size implies either that the DT result benefits from information the other models did not receive, that the DT evaluation/test split is contaminated, or that one of the reported tables is incorrect. The paper also does not state the train/test split (Section 4.1), does not restrict the correlation threshold (Section 3.2.1) or autoencoder fitting to the training partition, and provides no ablation without the autoencoder, so there is no independent evidence that the autoencoder is the cause of the 99.94% figure. Consequently the strongest claim, a lightweight autoencoder-based IDS at 99.94% F1, is not supported by the evidence as reported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript presents a lightweight intrusion detection system for IIoT networks based on an autoencoder that compresses 24 features into a 6-dimensional bottleneck using a cost-sensitive MSE loss, followed by a set of classifiers (Decision Tree, XGBoost, LightGBM, LDA, TabNet, LSTM). The authors report that the Decision Tree achieves 99.94% accuracy and F1 on the multiclass Edge-IIoTset task, and that the model can be deployed on a Jetson Nano with 0.187 ms per-sample inference in multiclass mode. The paper also claims novelty for being the first to deploy such a model on a Jetson Nano.","tokens_in":8649,"tokens_out":5373,"duration_ms":47499,"significance":"If the reported results were reproducible, the main contributions would be a demonstration that six autoencoder-derived features plus a decision tree yield near-perfect multiclass intrusion-detection performance on a realistic imbalanced IIoT benchmark, together with a practical edge-deployment data point. The use of the public Edge-IIoTset dataset is a strength for reproducibility, and the Jetson Nano inference measurements are useful practical evidence. However, the experimental protocol as written does not establish the central claim: no train/test split is stated, no baseline without the autoencoder is provided, and the multiclass results contain a large unexplained performance gap between the Decision Tree and the gradient-boosted tree models.","major_comments":[{"comment":"The Decision Tree is reported with 0.9994 accuracy and F1, while XGB and LGBM on the same autoencoder features achieve only 0.8181 and 0.7946. For tabular data, gradient-boosted trees generally match or surpass a single decision tree; an 18-20 point shortfall is not explained by any stated hyperparameter choice or class imbalance. This gap suggests the DT and XGB/LGBM evaluations may not have used the same test set, or that the DT result benefits from information leakage. The authors must provide the exact evaluation protocol, the test-set definition, and per-class results for all models.","section":"Section 4.5, Table 4"},{"comment":"No train/test split is specified anywhere in the paper. The manuscript does not state whether the correlation-threshold filtering (>0.6), autoencoder training, and class-weight computation were performed only on the training partition. If any of these steps used the full dataset, the reported 99.94% accuracy is not an unbiased estimate of generalization. The authors must report the split ratio, stratification strategy, and a clear statement that all preprocessing and feature learning used training data only.","section":"Section 4.1, Section 3.2.1, Section 3.3"},{"comment":"The claim that the autoencoder 'enhances feature learning and overall detection accuracy' is not supported by an ablation. There is no comparison of the selected classifiers on the original 24 features versus the 6 autoencoder features, and no sensitivity analysis for the bottleneck size or the correlation threshold. Without these controls, the reported 99.94% figure cannot be attributed to the autoencoder.","section":"Section 3.3, Section 4.5"},{"comment":"The confusion matrices are inconsistent with the dataset description. The counts in Table 2 sum to 30,440 instances, whereas Table 1 describes 1,927,304 records and no test-subset size is given. The authors should clarify the size and composition of the evaluation set and present confusion matrices whose counts can be reconciled with the stated data split.","section":"Section 4.3, Table 2"},{"comment":"The autoencoder is described only by input dimension and bottleneck dimension; the number of hidden layers, activation functions, optimizer settings, batch size, epochs, and early-stopping criteria are omitted. This makes the core contribution impossible to reproduce. The authors should provide these hyperparameters or a pointer to the exact implementation.","section":"Section 3.3, Algorithm 1"}],"minor_comments":[{"comment":"Table 1 reports a total of 1,927,304 records, while the text in Section 3.1 states 2.2 million; please reconcile these numbers.","section":"Section 3.1, Table 1"},{"comment":"The inference-time section does not state the number of test instances used, so the reader cannot verify the reported per-sample averages of 0.184 ms and 0.187 ms.","section":"Section 4.6"},{"comment":"The test time for TabNet is shown as '—'; please report it or state explicitly why it was not measured.","section":"Table 4"},{"comment":"For Ferrag et al., the columns list accuracy as '94.67' and F1 as '99'; please clarify whether these are percentages and unify the formatting with the other rows.","section":"Table 5"},{"comment":"The panels in Figure 4 are captioned 'Tabnet/Binary Loss curve' and 'BiLSTM/Binary Loss curve', but the surrounding text sometimes refers to autoencoder loss curves; please align the captions with the actual content.","section":"Figure 4"},{"comment":"The claim of being 'the first to deploy our model on a Jetson Nano' is a strong novelty statement that is difficult to verify; consider softening it to 'we report deployment on a Jetson Nano.'","section":"Abstract and Section 4.7"}],"recommendation":"major_revision","confidential_remarks":"The missing train/test split and the unexplained Decision Tree versus XGB/LGBM gap are serious problems. If the authors cannot provide a clean experimental protocol and reproduce the results under it, the paper should not be accepted. The revision must include a full ablation and per-class metrics to make the 99.94% claim credible."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take on arXiv:2501.15266. The headline number, 99.94% multiclass F1 for a decision tree on six autoencoder-compressed features, is not credible on the evidence presented. The paper's own boosted-tree baselines get 81.8% (XGB) and 79.5% (LightGBM) on the same features. If the bottleneck features were that class-separable, boosted trees would match or beat a single tree; an 18-20 point gap points to an evaluation error or information leakage, not a validated result. That inconsistency is the load-bearing problem.\n\nCredit where due: the authors do run a usable benchmark on Edge-IIoTset, a widely used public dataset, and report real inference latencies on a Jetson Nano. The sub-millisecond numbers for a decision tree are plausible, and that kind of edge deployment data is useful for engineers. The cost-sensitive autoencoder is standard inverse-frequency weighting, but it's implemented cleanly.\n\nThe soft spots are substantial. There's no stated train/test split; Table 2 implies a test set of roughly 30k samples while the dataset has 1.9M rows, and the subsampling is never described. No ablation compares the pipeline with and without the autoencoder, so the central claim that the autoencoder helps is unproven. No error bars or repeated trials. The 'first to deploy on Jetson Nano' claim has no citation and is likely false. And the comparison table shows several prior methods with equal or better accuracy, so the 99.94% isn't a new SOTA anyway.\n\nFor me this is a lightweight engineering report, not a research contribution. The methodology is textbook, the evaluation is underspecified, and the key result is internally inconsistent. I would not send it to peer review; I'd ask the authors to provide the split, add an ablation, and resolve the boosted-tree gap first.","headline":"The main accuracy claim is undermined by an internal inconsistency with the boosted-tree baselines, and the experimental write-up is too thin to rescue it.","tokens_in":9205,"tokens_out":5063,"would_cite":false,"duration_ms":40526,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a cost-sensitive autoencoder that shrinks IIoT traffic to six latent features lets a decision tree reach 99.94% accuracy and F1 on 14 attack classes, with 0.187 ms per-sample inference on a Jetson Nano.","keywords":["Industrial Internet of Things","Intrusion Detection System","Autoencoder","Dimensionality Reduction","Class Imbalance","Edge Computing","Edge-IIoTset","Decision Tree"],"falsifier":"Take the Edge-IIoTset data, fit every preprocessing step—the correlation filter, the autoencoder, and the class weights—only on a training partition, and evaluate the same Decision Tree on a completely untouched test partition; if the multiclass F1 falls materially below 99.94%, the reported number depends on test-set information rather than on the autoencoder alone.","tokens_in":8163,"feed_emoji":"🛡️","tokens_out":12821,"duration_ms":102231,"temperature":0.7,"pith_summary":"This paper is trying to establish that a simple, lightweight classifier can handle the class imbalance and multiclass complexity that make IIoT intrusion detection hard, as long as the raw features are first passed through a cost-sensitive autoencoder. The authors report that a decision tree trained on the autoencoder's six-dimensional bottleneck reaches 99.94% accuracy and F1 on the Edge-IIoTset dataset across normal traffic and 14 attack types. They also report binary and multiclass inference times of 0.185 ms and 0.187 ms per sample on a Jetson Nano. If these numbers hold, a low-power edge device could screen industrial network traffic in real time without sacrificing detection quality.","feed_headline":"Autoencoder-shrunk features catch 99.94% of IIoT attacks","feed_subtitle":"A decision tree on six learned features runs in 0.187 ms per sample on a Jetson Nano edge device.","key_machinery":"The central object is the cost-sensitive autoencoder: a deterministic feedforward network whose encoder maps 24 features to a six-unit bottleneck, $h = f(x) = \\sigma(Wx+b)$, and whose decoder reconstructs the input, $\\hat{x} = g(h) = \\sigma(W'h+b')$. The reconstruction loss is weighted by class frequency, $L = \\frac{1}{N}\\sum_{i=1}^{N} w_{y_i}(x_i - \\hat{x}_i)^2$, with $w_{y_i}$ larger for rare attack classes. That weighted bottleneck is what the downstream classifiers consume; the paper argues this is why a Decision Tree can reach 99.94% multiclass F1 without oversampling or data augmentation.","core_discovery":"On its own terms, the paper's discovery is that deterministic autoencoding with class-weighted reconstruction loss converts an imbalanced, multiclass IIoT traffic dataset into a compact latent space where a plain decision tree saturates performance. The autoencoder compresses 24 normalized features down to six bottleneck units; the training loss is the per-sample squared reconstruction error multiplied by a weight inversely proportional to the class frequency, which the authors say lets rare attacks like MITM and fingerprinting shape the representation. The best multiclass result is 99.94% accuracy and F1 for the Decision Tree, with binary classification at 99.98% accuracy and the lightweight design running at 0.185 ms (binary) and 0.187 ms (multiclass) per sample on a Jetson Nano. The paper also claims this is the first intrusion-detection deployment on a Jetson Nano and that the cost-sensitive weighting removes the need for oversampling or augmentation.","pith_inferences":["A strict reproducibility run with the correlation filter, bottleneck size, and class weights fit only on a training partition would settle whether the 99.94% number is a property of the method or of tuning on the full dataset.","The same 24-to-6 autoencoder could be tested on other imbalanced IIoT datasets such as MQTTset and WUSTL-IIoT-2021; if the bottleneck size must change per dataset, the method is a recipe rather than a fixed architecture.","Because the bottleneck has only six units, inspecting which input features activate each unit could turn the compressed representation into an explanation tool for why a flow is flagged as an attack."],"forward_implications":["If the results hold, most of the original feature set is redundant: 24 features compressed to 6 are enough for near-perfect detection of 14 attack types plus normal traffic.","Edge screening becomes plausible on low-power hardware: per-sample inference near 0.19 ms means thousands of flows per second on a single Jetson Nano.","The cost-sensitive autoencoder offers an alternative to oversampling and synthetic augmentation for imbalanced network datasets, reducing preprocessing complexity.","A simple Decision Tree can match or beat far heavier deep classifiers once the representation is learned, which lowers the compute and energy budget for deployed IDS."],"supporting_citations":[{"why":"Defines the Edge-IIoTset dataset used for all experiments and also supplies the DT/RF/SVM/KNN/DNN comparison baseline in Table 5.","marker":"[Ferrag et al., 2022]"},{"why":"Provides the SA-DCNN baseline at 99.95% accuracy and 99.53% F1 that the paper's 99.94% result is positioned against.","marker":"[Alshehri et al., 2024]"},{"why":"Provides the MAGRU baseline at 99.97% accuracy and 99.64% F1 that the paper claims to complement with edge deployment.","marker":"[Ullah et al., 2023]"},{"why":"Provides the decision-tree and gradient-boosting anomaly-detection baseline used to situate the proposed lightweight models.","marker":"[Douiba et al., 2023]"}],"fun_headline_variants":["99.94% IIoT attack detection on a six-feature autoencoder","Decision tree on autoencoder latents: 0.187 ms per IIoT sample","First Jetson Nano IDS: 99.94% accuracy, 0.185 ms inference","Cost-weighted autoencoder lifts rare IIoT attack detection to 99.94%","Lightweight IIoT IDS: autoencoder + tree hits 99.94% in 0.187 ms"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline accuracy is only as strong as the guarantee that the test set was never used to choose the correlation threshold, the six-feature bottleneck size, or the class weights; if those choices were tuned on the full dataset, the 99.94% reflects leakage rather than generalization.","fun_headline_variants_meta":{"raw":{"variants":["99.94% IIoT attack detection on a six-feature autoencoder","Decision tree on autoencoder latents: 0.187 ms per IIoT sample","First Jetson Nano IDS: 99.94% accuracy, 0.185 ms inference","Cost-weighted autoencoder lifts rare IIoT attack detection to 99.94%","Lightweight IIoT IDS: autoencoder + tree hits 99.94% in 0.187 ms"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000549,"raw_usage":{"total_tokens":2643,"prompt_tokens":986,"completion_tokens":1657,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":602,"completion_tokens_details":{"reasoning_tokens":1540}},"tokens_in":602,"tokens_out":1657,"duration_ms":10766,"temperature":1.0,"reasoning_tokens":1540,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T14:26:24.621600+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the Edge-IIoTset data, fit every preprocessing step—the correlation filter, the autoencoder, and the class weights—only on a training partition, and evaluate the same Decision Tree on a completely untouched test partition; if the multiclass F1 falls materially below 99.94%, the reported number depends on test-set information rather than on the autoencoder alone.","supporting_citations":[{"cited_title":"A., Friha, O., Hamouda, D., Maglaras, L., and Janicke, H","cited_arxiv_id":null,"evidence_quote":"Defines the Edge-IIoTset dataset used for all experiments and also supplies the DT/RF/SVM/KNN/DNN comparison baseline in Table 5."},{"cited_title":"S., Saidani, O., Alrayes, F","cited_arxiv_id":null,"evidence_quote":"Provides the SA-DCNN baseline at 99.95% accuracy and 99.53% F1 that the paper's 99.94% result is positioned against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the MAGRU baseline at 99.97% accuracy and 99.64% F1 that the paper claims to complement with edge deployment."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the decision-tree and gradient-boosting anomaly-detection baseline used to situate the proposed lightweight models."}],"review_version":1}