{"id":"b16e2678-1e46-4793-b411-14ee1a2f262c","arxiv_id":"2504.14122","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"An ensemble of LSTM, GRU, and stacked autoencoders trained only on normal web requests is reported to detect zero-day web attacks with 97.58 percent accuracy and a 0.2 percent false-positive rate on CSIC2012.","lead":"This paper combines three neural network autoencoders to flag web requests that look different from normal traffic, aiming to catch previously unseen attacks. The authors report 97.58 percent accuracy and a 0.2 percent false-positive rate on a standard web attack dataset, but the evaluation has serious statistical gaps.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline metrics are unsupported because the MAE threshold is selected on the evaluation data and the test set is 97.5% malicious, making accuracy and precision prevalence-driven.","rationale":"The paper's strongest claim is entirely empirical, and there is no formal verification, released code, or error analysis to provide independent support. The experiment's validity hinges on two conditions: the detection threshold must be chosen without using the test labels, and the reported metrics must not be distorted by extreme class imbalance. Both conditions fail in the submitted manuscript. Section 5 states that the threshold of 4.09 was obtained by iterative experimentation and inspection of the MAE density, which is exactly the kind of post hoc fitting that invalidates the FPR and recall estimates. The test set in Table 2 contains 50,174 malicious versus 1,299 normal requests, so accuracy and precision are almost completely determined by the prevalence of attacks in the test set rather than by model skill. The class-conditioned metrics, recall and specificity, are more informative, but they are only meaningful if the threshold is fixed in advance. The reader's weakest assumption identified the same general problem: the evaluation does not reflect a realistic zero-day deployment with a separate threshold-selection procedure. My concern sharpens that by showing the threshold leakage is the more fundamental flaw, with class imbalance as an aggravating factor that renders the headline accuracy and precision uninterpretable. The dataset count discrepancy between Section 5.1 and Table 2 adds further uncertainty about what was actually evaluated. Because the central claim is the measured performance and that performance is not independently estimable from the described protocol, the REJECT verdict is appropriate, and my analysis does not call for a change in the reader's verdict.","tokens_in":16474,"tokens_out":3427,"duration_ms":32988,"concrete_test":"Use a temporal or random split of CSIC2012: train the autoencoders on 80% of normal requests only; tune the MAE threshold on a separate validation set of normal requests plus a small labeled attack subset (or a synthetic attack set), not on the final test set; then evaluate recall on the held-out malicious requests and FPR on the held-out normal requests. Also compute precision under a realistic deployment prior, e.g., 90% normal and 10% attack. If the threshold selected on validation differs materially from 4.09, if balanced recall or FPR changes, or if precision at realistic prevalence is far below 99.99%, the reported claims do not transfer.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is a low-FPR zero-day anomaly detector, but the evaluation protocol cannot support it. In Section 5, after Eq. (7), the decision threshold is determined by inspecting the MAE density (Figure 6) and via 'iterative experimentation' on the same data that later produces Table 3; no separate validation split is described. This makes the reported 0.2% FPR and 97.52% recall partly a selection result, not an unbiased estimate. The test-set composition compounds the problem: Table 2 lists 50,174 malicious and 1,299 normal requests, so accuracy (0.9758) is barely above an always-malicious baseline (0.9748), and precision (0.9999) is mostly a prevalence artifact. Nothing in the paper shows the model would maintain recall or FPR if the threshold were fixed on a held-out validation set and evaluated in traffic with realistic normal-majority prevalence. The dataset description also conflicts: Section 5.1 says roughly 16,000 labeled anomalies, while Table 2 uses 50,174 positives, so the provenance of the test set is unclear. These issues are nested: the headline numbers may be true on this particular split and threshold, but they do not establish generalization to zero-day traffic.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an ensemble of LSTM, GRU, and stacked autoencoders for detecting zero-day web attacks, trained exclusively on normal web requests from the CSIC2012 dataset. Input requests are tokenized at word level, mapped to numeric sequences, encoded and decoded by the three autoencoders, and the concatenated latent representations are compressed through a dense layer. Anomaly detection is performed by thresholding the mean absolute error (MAE) between the reconstructed and original request. The authors report accuracy 97.58%, recall 97.52%, specificity 99.76%, precision 99.99%, and a false positive rate of 0.2%, and claim superiority over each individual sub-model and over prior CSIC-based detectors.","tokens_in":16717,"tokens_out":5348,"duration_ms":45871,"significance":"If the reported evaluation were valid, the proposed method would be a practically attractive low-FPR anomaly detector for web traffic, and the tokenization-plus-ensemble-compression idea would be a useful design pattern. The authors also deserve credit for explicitly reporting FPR, which many prior web-attack detection studies omit. However, the evaluation protocol as presented does not support the headline claims because the classification threshold is selected on the same data used to compute the metrics, and because the test set is so imbalanced that accuracy and precision are dominated by the majority class.","major_comments":[{"comment":"The test set contains 50,174 malicious and only 1,299 normal requests, so the reported accuracy of 0.9758 is only 0.10 percentage points above the always-malicious baseline of 50,174/51,473 = 0.9748. Similarly, the 99.99% precision is largely a consequence of the 97.5% prior probability of the malicious class. The central claim of 'remarkable detection metrics' is therefore prevalence-driven; the paper should report balanced metrics such as balanced accuracy, PR-AUC, and metrics evaluated under a realistic normal-majority test distribution, and should compare explicitly against the majority-class baseline.","section":"5.3, Table 2"},{"comment":"The classification threshold (approximately 4.09) is determined by inspecting the MAE density plot (Figure 6) and by 'iterative experimentation' on the same data that is later used to produce Table 3. No separate validation split is described for threshold selection. Consequently, the reported FPR of 0.2% and recall of 97.52% are fitted quantities rather than unbiased performance estimates. The authors should select the threshold on a held-out validation set (or via nested resampling) and report the resulting test metrics.","section":"5, after Eq. (7) and Figure 6"},{"comment":"The text states that 'The dataset comprises approximately 16,000 instances labeled as anomalous,' but Table 2 reports 50,174 malicious requests in the test set and a total of 51,473 requests. This inconsistency makes the provenance of the test set unclear and prevents reproducibility. The authors must clarify the exact data split, how the anomalous samples were selected, and reconcile the anomaly count with the reported table.","section":"5.1 vs. Table 2"},{"comment":"The paper states that 'The reported results represent the average performance obtained over six independent runs of the model,' but Table 3 reports only point estimates with no variance, and no statistical comparison is made against the sub-models. Since the FPR difference between the ensemble (3 false positives) and the LSTM/GRU sub-models (14 false positives) rests on small counts, the authors should report the distribution of metrics across runs and show whether the ensemble advantage is stable.","section":"5.3"}],"minor_comments":[{"comment":"The sentence 'this method addresses the challenges of training neural network-based models for web security, which arise from the inherent variability in the length and structure of web requests' is repeated almost verbatim in consecutive sentences; please remove the duplication.","section":"4.1.1"},{"comment":"The approach is described as 'unsupervised' in Section 2 but as a 'one-class' method in the Abstract; please use consistent terminology.","section":"2 and Abstract"},{"comment":"Table 1 lists MAE as the 'Evaluation Metric for measuring prediction accuracy,' but MAE is a reconstruction error used for anomaly scoring, not an accuracy metric; please reword.","section":"Table 1"},{"comment":"The related work states that Ingham et al. [9] uses 'Transformer models,' but the cited bibliographic entry is 'Learning DFA representations of HTTP' (Computer Networks, 2007); the citation appears mismatched.","section":"3, reference [9]"},{"comment":"Please report all hyperparameters for reproducibility, including the exact number of units in the final dense compression layer, the sequence padding length, and the random seeds used; the current architecture description omits some of these details.","section":"5.2"},{"comment":"The caption relies on color ('blue points', 'red line'); please ensure the figure is legible in grayscale or add markers.","section":"Figure 9"}],"recommendation":"major_revision","confidential_remarks":"The reader's report and the stress-test note correctly identify the two central problems: threshold selection on the evaluation data and the extreme class imbalance. The dataset inconsistency (16,000 vs. 50,174 anomalies) is particularly concerning because it suggests the authors may not have full control over their data pipeline. These issues are correctable in principle with a re-run evaluation using a held-out validation set and a more realistic test distribution, but if the authors cannot provide such an evaluation, the paper should not be accepted. The related-work section also lists many references with limited critical engagement; several citations appear tangential to the proposed method."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a plausible incremental architecture paper whose headline numbers do not survive contact with its own evaluation. The ensemble of LSTM, GRU, and stacked autoencoders with concatenated latent codes compressed through a dense layer is a reasonable combination of known components, and the character-class tokenization is a sensible way to normalize HTTP requests. Training only on normal traffic and explicitly reporting FPR is the right frame for zero-day detection. I also give the paper credit for comparing the ensemble against each sub-model on the same benchmark. What is missing is an ablation, so we cannot attribute the gain to any particular design choice, but the idea is not silly.\n\nThe evaluative claim is the soft spot, and it is load-bearing. Table 2 shows 50,174 malicious requests against 1,299 normal requests. An always-malicious classifier would score 97.48% accuracy, barely below the reported 97.58%; the 99.99% precision and 0.2% FPR are mostly prevalence artifacts. The threshold of 4.09 is chosen by inspecting Figure 6 and 'iterative experimentation' on the same data later used to produce Table 3, with no separate validation split described. That makes the reported FPR and recall selected numbers, not unbiased estimates. There is also a stated contradiction: Section 5.1 says roughly 16,000 anomalies, while Table 2 uses 50,174 positives. And although the authors say results are averaged over six runs, no variance or error bars appear anywhere. None of this is fatal to the underlying approach, but it is fatal to the paper's central claim as presented.\n\nWho is this for? A reader looking for architecture ideas in one-class web anomaly detection might skim the method section. A reader looking for reliable evidence on zero-day detection should not rely on these results. The paper is not incoherent or unserious; it is an unevaluated model in search of a valid experiment. I would not cite the headline metrics and would not send this to review in its current form. My recommendation to the editor: desk reject, with encouragement to resubmit after a clean evaluation—balanced or at least normal-majority test split, threshold chosen on validation data only, simple baselines, and released code.","headline":"The ensemble idea is reasonable, but the reported 97.58% accuracy is mostly a prevalence artifact of a 97.5%-malicious test set, and the decision threshold is tuned on the same data used to report the headline FPR.","tokens_in":17240,"tokens_out":3049,"would_cite":false,"duration_ms":26650,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An ensemble of LSTM, GRU, and stacked autoencoders, trained only on normal web requests, is claimed to detect zero-day web attacks with 97.58% accuracy, 99.76% specificity, and a 0.2% false-positive rate on CSIC2012.","keywords":["zero-day web attacks","anomaly detection","autoencoder ensemble","LSTM autoencoder","GRU autoencoder","stacked autoencoder","tokenization","CSIC2012"],"falsifier":"Run the trained ensemble on a naturally balanced or held-out set of HTTP requests without re-tuning the 4.09 MAE threshold, and record the fraction of normal requests whose reconstruction error exceeds the threshold; if that fraction is substantially above 0.2% or recall falls well below 97.5%, the zero-day detection claim as stated does not generalize.","tokens_in":16254,"feed_emoji":"🛡️","tokens_out":5206,"duration_ms":44419,"temperature":0.7,"pith_summary":"This paper aims to show that an ensemble of three autoencoders, LSTM, GRU, and stacked, trained exclusively on normal web requests, can detect zero-day web attacks such as SQL injection, cross-site scripting, and buffer overflow without ever seeing an attack sample. The authors report 97.58% accuracy, 97.52% recall, 99.76% specificity, and 99.99% precision on the CSIC2012 benchmark, with a false-positive rate of only 0.2%. A reader should care because a detector that needs no attack labels can be pointed at novel threats with minimal retraining, and a very low false-positive rate means legitimate users are rarely blocked. The evidence comes from six repeated runs on CSIC2012, comparing the ensemble against each sub-model and against earlier CSIC-based detectors.","feed_headline":"Zero-day web attacks caught with 0.2% false-positive rate","feed_subtitle":"Trained only on normal traffic, three autoencoders catch unseen SQL injection, XSS, and buffer-overflow requests.","key_machinery":"The load-bearing mechanism is a one-class reconstruction ensemble. Each sub-model is a four-layer autoencoder with two encoder layers of 50 and 25 units and two symmetric decoder layers, so each compresses a tokenized request into a latent vector. The innovation is that the three latent vectors are not averaged or majority-voted; they are concatenated and squeezed through a 50-unit dense layer, forcing the ensemble to keep only mutually informative features. At test time the decoded request's Mean Absolute Error ($\\mathrm{MAE}=\\frac{1}{n}\\sum_i |\\hat{x}_i-x_i|$) is thresholded, and values above the empirically selected threshold of roughly 4.09 are flagged as malicious. The character-class tokenizer is what turns variable-length raw requests into compact, structured numeric sequences the autoencoders can reconstruct.","core_discovery":"The central claim is that zero-day web attack detection can be treated as a one-class reconstruction problem: the model learns the distribution of normal requests only, and any request whose reconstruction error is too large is declared malicious. The proposed pipeline tokenizes each request by classifying characters into classes such as numeric, lowercase, uppercase, and special, maps the resulting token sequence to numbers, and feeds it to three autoencoders in parallel. Their latent vectors are concatenated and then compressed through a dense layer, and the reconstruction error, measured as Mean Absolute Error, is compared with a threshold of about 4.09. On CSIC2012 the ensemble reports a false-positive rate of 0.2%, meaning 3 of 1,299 normal requests are misclassified, while catching 48,934 of 50,174 malicious requests.","pith_inferences":["The headline accuracy mostly reflects that 50,174 of 51,473 test requests are malicious, so the number that matters in deployment is the 0.2% false-positive rate, and that is the claim most worth stress-testing.","A natural extension is to run the fixed threshold of 4.09 on requests from a different dataset such as HTTPParams or FWAF without retuning; if the false-positive rate climbs, the model is overfit to CSIC2012's request grammar rather than to normality in general.","The concat-and-compress ensemble suggests a general recipe: train several cheap one-class reconstructors on the same normal data and let a learned compression layer reconcile their disagreements, rather than averaging their scores.","Because the threshold appears to be selected after experimenting on the same data that is later scored, a deployment would need the threshold fixed on separate validation traffic before seeing attack data to know whether the metrics transfer."],"forward_implications":["Because no attack labels are needed for training, the same pipeline can be retargeted to a new web application by collecting only its normal request logs.","At the reported false-positive rate, only about 0.2% of legitimate requests would be blocked, which is low enough to deploy in front of real users without constant manual review.","The concatenate-and-compress ensemble achieves better accuracy, recall, and false-positive rate than any of the three autoencoders alone, suggesting that combining diverse one-class representations is the source of the gain.","With roughly 20 seconds of training and 5 seconds of testing on the reported setup, the method is computationally light enough for inline or near-real-time deployment."],"supporting_citations":[{"why":"Supplies the CSIC2010 and CSIC2012 benchmark datasets used for training and evaluation.","marker":"[42]"},{"why":"The GitHub CNN WAF project from which the CSIC2012 dataset was sourced and preprocessed.","marker":"[43]"},{"why":"Earlier stacked-autoencoder anomaly detection for web attacks that motivates the one-class autoencoder design.","marker":"[12]"},{"why":"Prior RNN-based anomaly detection using LSTM or GRU units and tokenized request patterns, whose tokenization approach is adapted.","marker":"[14]"},{"why":"An encoder-decoder recurrent model for zero-day web attacks that defines the direct problem the proposed ensemble targets.","marker":"[16]"},{"why":"A one-class GRU autoencoder combined with ensemble learning for anomaly detection, a conceptual template for one-class autoencoder ensembles.","marker":"[32]"}],"fun_headline_variants":["Ensemble autoencoders spot zero-day attacks with 0.2% false positives","LSTM+GRU+stacked autoencoders catch zero-day attacks at 0.2% FPR","Autoencoder ensemble detects unseen web attacks with 99.99% precision","One-class autoencoder trio nails zero-day web attacks, 0.2% FPR","Novel tokenizer plus autoencoder trio catches unknown web attacks at 0.2% FPR"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline figures rest on a test set where 50,174 of 51,473 requests are malicious and on a reconstruction-error threshold chosen by experimenting on that same data, so the 0.2% false-positive rate may not transfer to a real traffic mix or an independently chosen threshold.","fun_headline_variants_meta":{"raw":{"variants":["Ensemble autoencoders spot zero-day attacks with 0.2% false positives","LSTM+GRU+stacked autoencoders catch zero-day attacks at 0.2% FPR","Autoencoder ensemble detects unseen web attacks with 99.99% precision","One-class autoencoder trio nails zero-day web attacks, 0.2% FPR","Novel tokenizer plus autoencoder trio catches unknown web attacks at 0.2% FPR"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001243,"raw_usage":{"total_tokens":5106,"prompt_tokens":955,"completion_tokens":4151,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":571,"completion_tokens_details":{"reasoning_tokens":4033}},"tokens_in":571,"tokens_out":4151,"duration_ms":25753,"temperature":1.0,"reasoning_tokens":4033,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:56:04.135825+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the trained ensemble on a naturally balanced or held-out set of HTTP requests without re-tuning the 4.09 MAE threshold, and record the fraction of normal requests whose reconstruction error exceeds the threshold; if that fraction is substantially above 0.2% or recall falls well below 97.5%, the zero-day detection claim as stated does not generalize.","supporting_citations":[{"cited_title":"Web application firewall using machine learning and features engineering","cited_arxiv_id":null,"evidence_quote":"Supplies the CSIC2010 and CSIC2012 benchmark datasets used for training and evaluation."},{"cited_title":"CNN Web Application Firewall","cited_arxiv_id":null,"evidence_quote":"The GitHub CNN WAF project from which the CSIC2012 dataset was sourced and preprocessed."},{"cited_title":"An anomaly detection method to detect web attacks using stacked auto-encoder","cited_arxiv_id":null,"evidence_quote":"Earlier stacked-autoencoder anomaly detection for web attacks that motivates the one-class autoencoder design."},{"cited_title":"Anomaly-based web attack detection: a deep learning approach","cited_arxiv_id":null,"evidence_quote":"Prior RNN-based anomaly detection using LSTM or GRU units and tokenized request patterns, whose tokenization approach is adapted."},{"cited_title":"Zerowall: Detecting zero-day web attacks through encoder-decoder recurrent neural networks","cited_arxiv_id":null,"evidence_quote":"An encoder-decoder recurrent model for zero-day web attacks that defines the direct problem the proposed ensemble targets."},{"cited_title":"A lightweight intelligent network intrusion detection system using one-class autoencoder and ensemble learning for IoT","cited_arxiv_id":null,"evidence_quote":"A one-class GRU autoencoder combined with ensemble learning for anomaly detection, a conceptual template for one-class autoencoder ensembles."}],"review_version":1}