{"id":"2c9eda3e-24cb-4a9e-9625-067422d57c8c","arxiv_id":"1908.01928","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"An unsupervised system-call monitoring system using LSTM detects recreated web exploits with AUC up to 0.97 and generally beats PCA and one-class SVM baselines across four applications.","lead":"This paper presents AppMine, a system that watches the system calls of web applications running in Docker containers and flags unusual behavior as possible attacks. It reports that a neural network model detects recreated exploits, including the Apache Struts flaw behind the Equifax breach, with higher accuracy than two standard baselines.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"LSTM advantage may be an artifact of input representation: baselines see 1-second histograms while LSTM sees 15-second sequences, so the claimed temporal-dependency benefit is not isolated.","rationale":"The reader's weakest assumption concerns the realism of synthetic legitimate workloads and transfer to production traffic. That is a legitimate external-validity concern, but the most load-bearing issue for the paper's central claim is internal: the evaluation does not isolate the effect of sequence modeling. The LSTM is given 15 one-second windows of history while PCA and OCSVM are given a single one-second vector. Any model with access to a longer context could plausibly outperform models without it, regardless of architecture. The paper's design insight (Section 2.3) explicitly motivates the LSTM by the insufficiency of methods applied 'independently of their temporal ordering,' yet the traditional baselines are exactly such methods, with no attempt to provide them with temporal context. Thus the reported AUC gaps do not establish that LSTM's recurrent structure is the cause; they could be due to the feature window. A second internal issue is the selection of the time-window length based on ROC curves, which leaks test labels into model configuration; this could bias all three models' reported performance upward and makes the absolute AUC values untrustworthy. These concerns are concrete and testable: a straightforward experiment giving the baselines equivalent context would settle whether the LSTM advantage is real. The paper remains a useful empirical study, and the conditional verdict is appropriate; the authors should add this ablation and clarify their model-selection procedure before the central claim is accepted.","tokens_in":14730,"tokens_out":4040,"duration_ms":43721,"concrete_test":"Recompute the Struts CVE-2017-5638 comparison with PCA and OCSVM trained on the same 15-window input representation used by the LSTM (e.g., concatenate or average the 15 one-second frequency vectors into a single feature vector, or use a 15-second window), keeping identical train/test splits and threshold selection. If PCA/OCSVM AUCs rise to within 0.05 of LSTM's 0.97, the claimed advantage is an artifact of feature representation; if they remain near 0.81/0.83, the temporal-modeling claim gains support. Also report results with the window length fixed before any test ROC inspection.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim attributes LSTM's higher AUC to its ability to model sequences of system calls over time (Section 2.3, Section 3.4). But the comparison confounds model architecture with input representation: PCA and OCSVM receive a single 1-second frequency vector (Section 3.4), while the LSTM receives a sequence of 15 such vectors (Table 4). The baselines are therefore denied access to any temporal or contextual information, so the reported gap (e.g., 0.97 for LSTM vs 0.81/0.83 for PCA/OCSVM on Struts CVE-2017-5638) may reflect the longer context window rather than the LSTM's sequential modeling. The paper gives no ablation or baseline with equivalent context, and its own Figure 7(d) shows OCSVM beating LSTM on one application, undermining the blanket claim of improvement. Additionally, the 1-second window length was chosen after inspecting ROC curves (Section 4), which uses test labels for model selection and could inflate all reported AUCs. Both issues bear directly on whether the LSTM's advantage is real and attributable to temporal dependencies.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents AppMine, an unsupervised anomaly detection system for containerized web applications. The system collects system-call frequency vectors from Docker containers via Sysdig, then trains either PCA, one-class SVM, or an LSTM on legitimate data only. The LSTM consumes a sequence of 15 one-second frequency vectors and predicts the next vector, with an inverse-frequency-weighted distance used as the anomaly score. On a testbed with four web applications and seven recreated Metasploit exploits (15 post-exploitation scripts), the paper reports that the LSTM model outperforms the traditional baselines, with Struts CVE-2017-5638 average AUC of 0.97 versus 0.81 for PCA and 0.83 for OCSVM. The main claims are that unsupervised, sequence-aware modeling of system-call behavior can detect previously unseen exploit activity, and that the LSTM architecture provides a significant advantage over frequency-vector-only anomaly detection.","tokens_in":14977,"tokens_out":4056,"duration_ms":83502,"significance":"If the reported results hold, the paper would provide a useful data point that deep sequence models can detect exploit activity in containerized web applications without attack data for training. The testbed is carefully constructed, with separate legitimate and attack collection, and the evaluation uses standard AUC metrics. The TF-IDF-weighted prediction-error distance for the LSTM is a sensible, if simple, contribution. The paper also clearly states the threat model. However, the central claim of LSTM superiority is currently confounded by an input-representation mismatch and by hyperparameter selection on labeled test data, and the 'improves by 0.09 to 0.25' statement is contradicted by the paper's own Figure 7 for two of seven configurations. These issues are fixable but require additional experiments and textual qualification; the current version overstates the evidence.","major_comments":[{"comment":"The comparison between the LSTM and the traditional baselines is confounded with input representation. PCA and OCSVM are said to take 'the system call frequency feature vector computed for one time interval' (one 1-second vector), while the LSTM receives a sequence of 15 such vectors (Table 4). Any AUC gain may therefore come from a longer context window rather than from the recurrent architecture's sequential modeling. The central claim that the LSTM 'exploit[s] the dependencies in sequences of system calls over time' would be supported by an ablation in which PCA/OCSVM receive the same 15-second context (e.g., a concatenated 15-vector input) or a feedforward network is trained on the same input. Without such a control, the attributed benefit of the LSTM is not isolated.","section":"Section 3.4, Table 4"},{"comment":"The selection of the 1-second time window is made after comparing ROC curves for 100ms, 500ms, 1s, and 2s, with the text stating that the other options were worse because 'the ROC curves were closer to the diagonal.' ROC curves require the attack labels on the test set, so this is a form of test-set-based hyperparameter selection. It can inflate the reported AUCs for all models and could also influence the relative ranking. The authors should either select hyperparameters using only training/validation data (for example, by a criterion on legitimate data such as reconstruction error or prediction error on a held-out legitimate set) or report the AUCs for all models under each candidate window length, so that the sensitivity of the comparison is visible.","section":"Section 4"},{"comment":"The statement that 'The AUC for LSTM is between 0.75 and 0.97 and improves the traditional models' average AUC between 0.09 and 0.25' is not supported by the paper's own aggregated results. In Figure 7(a) (Drupal), OCSVM achieves AUC 0.95 versus LSTM 0.93; in Figure 7(d) (WP Ajax Load More), OCSVM achieves AUC 0.86 versus LSTM 0.81. These are not small-gap outliers but direct counterexamples to the stated improvement range. The abstract and introduction should be qualified to reflect the configurations where LSTM outperforms the baselines, and the authors should discuss why the LSTM underperforms on those applications, for example in terms of workload characteristics or attack profiles.","section":"Figure 7 and Section 4, 'Comparison of LSTM with traditional models'"}],"minor_comments":[{"comment":"The caption of Figure 6 says 'ROC curves for enum_system for Struts with CVE-2017-5638,' but the text in Section 4 refers to 'the attack script enum_network' and the plot legend reads 'enum_network.' The caption should be corrected to match the actual plotted script.","section":"Figure 6"},{"comment":"The definition of the inverse-frequency weight f-bar_i in the weighted distance formula is not fully specified; the paper should state the exact formula used (for example, log(N/df_i) or a smoothed variant) and how it is computed from the training data.","section":"Section 3.4, LSTM training"},{"comment":"The hyperparameters for OCSVM (kernel type, nu, gamma) are not reported, although PCA components and all LSTM settings are given. These parameters materially affect OCSVM's AUC and should be listed for reproducibility.","section":"Section 4, hyperparameters"},{"comment":"The evaluation appears to be based on a single train/test split; reporting standard deviations or performing multiple random splits would strengthen the comparison and help assess whether the observed AUC differences are stable.","section":"Section 4"},{"comment":"There are minor typographical errors: Section 5 has 'Bayesian earning methods' instead of 'Bayesian learning methods,' and Section 6 has 'web application' where the plural is intended. The Swaddler description also uses 'worflow' instead of 'workflow.'","section":"Section 5 and Section 6"},{"comment":"The legitimate workloads are generated by Interactor (a Selenium-based simulator) and ftpbench; the paper should more explicitly discuss how the models might behave under noisier production workloads (e.g., variation in user concurrency or periodic administrative activity) and whether the reported false-positive rates would be expected to transfer.","section":"Section 3.1"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely and practical problem, but the evidence for the headline claim needs substantial strengthening. The confound between architecture and input representation is the most serious issue: the current comparison does not show that sequential modeling per se helps. The test-set-based window selection is also a methodological concern that a careful reviewer will catch. I would not reject the paper, because the testbed is useful and the question is worth answering, but the revision should include an ablation or at least a strong argument for why the input-representation difference does not explain the results. I would also ask the authors to reconcile the Drupal and WP Ajax results with the 'improves by 0.09 to 0.25' claim before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my read. The paper reports a real build: four Dockerized web apps, seven Metasploit exploits, 15 post-exploit scripts, Sysdig traces, and a comparison of unsupervised models on syscall frequency vectors. That testbed is the most useful part of the work. The training/evaluation setup is mostly sound: attack data held out, thresholds set on legitimate validation, and there is no circular derivation. The TF-IDF weighted error for the LSTM is a small but sensible idea. The paper is clearly written and honest that real traffic data would be better.\n\nThe soft spot is the headline claim. PCA and OCSVM are given a single 1-second histogram; the LSTM is given a sequence of 15 such vectors. So the LSTM's advantage could just be that it sees more context, not that its recurrent structure is better at modeling dependencies. There is no ablation feeding the baselines the same 15-second window, or feeding the LSTM a single window. Without that, saying 'LSTM exploits sequential dependencies' is not supported. The text also says LSTM improves average AUC by 0.09-0.25, but Figure 7 shows OCSVM at 0.95 vs LSTM 0.93 for Drupal, and OCSVM 0.86 vs LSTM 0.81 for WP Ajax Load More. Those are worse, not better. The claim needs tempering to 'usually better on this testbed.'\n\nThere is also a test-set leakage issue: the 1-second window length was chosen by looking at ROC curves, which use attack labels. That can inflate all reported AUCs and makes the absolute numbers unreliable. It is fixable by doing the selection on a separate validation split. Minor issues: test sets are small (13-28 minutes), no confidence intervals or significance tests, and no artifacts released.\n\nWho wants this? People working on host/container anomaly detection and anyone building benchmarks for web exploit detection. It is an incremental advance over DeepLog and Dymshits et al., but the testbed itself has value. I would send it to peer review with major revision, mainly to fix the baseline comparison and tone down the claims. It is not ready as-is.","headline":"A solid new testbed with unsupervised syscall anomaly detection, but the LSTM-vs-baseline comparison is confounded with context window and the headline improvement claim is overbroad.","tokens_in":15449,"tokens_out":3427,"would_cite":false,"duration_ms":30607,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"AppMine shows that an unsupervised LSTM, trained only on legitimate system-call sequences, detects web exploits that PCA and one-class SVM miss.","keywords":["web application security","anomaly detection","LSTM","system call analysis","Docker containers","unsupervised learning","intrusion detection","CVE-2017-5638"],"falsifier":"Run AppMine's trained models on a day of real production traffic from the same applications, with routine housekeeping jobs such as backups, cron jobs, and log rotation labeled as legitimate, and measure the false positive rate; if those bursts of file and network system calls resemble the tested post-exploitation scripts, the reported AUC at fixed false-positive rates would fall sharply.","tokens_in":14536,"feed_emoji":"🛡️","tokens_out":7275,"duration_ms":60636,"temperature":0.7,"pith_summary":"AppMine claims that an offline, unsupervised detector can catch web-application exploits by watching the sequence of system calls a containerized app makes, after being trained only on legitimate traffic. On a testbed of four web applications and seven recreated exploits, its LSTM model detects the Apache Struts CVE-2017-5638 exploit with an average AUC of 0.97, compared with 0.81 for PCA and 0.83 for one-class SVM. The value of the claim is that it needs no attack samples for training and so targets unknown vulnerabilities, the scenario patching and signature systems cannot cover.","feed_headline":"LSTM trained on normal traffic spots web exploits PCA and OCSVM miss","feed_subtitle":"AppMine learns normal system-call sequences in Docker apps and flags deviations, scoring 0.97 AUC on the Struts exploit.","key_machinery":"The load-bearing object is an LSTM sequence predictor over system-call frequency vectors. Each one-second window is represented as a vector counting how often each system call appears; the network takes a sequence of length 15, predicts the next vector, and the anomaly score is a distance between prediction and actual vector. The distance is weighted by each system call's inverse frequency in training, an adaptation of TF-IDF, which the paper shows separates the attack-distance distribution from the legitimate distribution far better than uniform weighting. This mechanism is what lets the model exploit dependencies over time rather than just per-window counts.","core_discovery":"The paper's central discovery is that the temporal ordering of system-call frequency vectors carries most of the signal for distinguishing an exploited web application from a healthy one. Where PCA and OCSVM, which examine each one-second window of system-call counts in isolation, achieve AUCs of 0.81 and 0.83 on the Struts CVE-2017-5638 exploit, an LSTM that consumes the preceding 15 one-second frequency vectors and predicts the next vector reaches 0.97. The same pattern holds across most of the fifteen post-exploitation scripts on four applications, with LSTM average AUCs between 0.75 and 0.97. The authors argue this is because attack activity differs from legitimate use mainly in the transition structure between windows, not in any single window's histogram.","pith_inferences":["A natural extension the authors do not run is to test the same detector on legitimate bursts that mimic post-exploitation scripts, such as scheduled backups, package-manager runs, or admin commands, since those could produce false positives that the synthetic legitimate workload never exercises.","The TF-IDF weighting suggests that much of the model's power may come from a few rare system calls; an ablation removing the rarest calls would reveal whether the detector is robust to call-set drift across kernel versions.","If the sequencing signal transfers, a similar framework could monitor containerized databases or message queues, not just web front-ends, as long as a clean legitimate workload can be generated."],"forward_implications":["An attacker who compromises a web application and runs the fifteen post-exploitation scripts tested produces a detectable deviation in the sequence of system-call counts even though individual windows may look normal.","Because training uses only legitimate data, the same pipeline can be pointed at a new application or zero-day exploit without re-labeling attack data, as long as clean legitimate logs exist.","A per-application threshold chosen on a legitimate validation set keeps false positives low at deployment time, and the LSTM's average AUC advantage over PCA and OCSVM on the testbed ranges from 0.09 to 0.25.","The sequencing benefit is not uniform: on Drupal and one WordPress plugin, OCSVM posted slightly higher average AUC than LSTM, so the approach's advantage depends on the application."],"supporting_citations":[{"why":"Supplies the system-call count vector representation used to build per-window features.","marker":"[8]"},{"why":"Foundational result that system-call sequences deviate under attack, motivating the whole approach.","marker":"[10]"},{"why":"Defines the LSTM architecture that the model uses to capture temporal dependencies.","marker":"[12]"},{"why":"Provides the exploit and post-exploitation modules used to generate attack data for evaluation.","marker":"[23]"},{"why":"Supplies the browser automation behind the Interactor tool that generates legitimate user workloads.","marker":"[33]"},{"why":"The container monitoring agent that records the system-call traces feeding the machine-learning models.","marker":"[37]"}],"fun_headline_variants":["System-call sequence LSTM beats PCA and OCSVM on web exploits","Temporal patterns in system calls reveal web attacks better than histograms","AppMine uses LSTM on system-call sequences to catch exploits others miss","Unsupervised LSTM on Docker system calls detects web vulnerabilities with 0.97 AUC","Time-aware anomaly detection for web apps outperforms static-window methods"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes that the synthetic legitimate workloads (browser sessions and FTP benchmark traffic) are representative enough of real user traffic that a model trained on them will transfer, and that normal production traffic never contains system call patterns similar to post-exploitation scripts.","fun_headline_variants_meta":{"raw":{"variants":["System-call sequence LSTM beats PCA and OCSVM on web exploits","Temporal patterns in system calls reveal web attacks better than histograms","AppMine uses LSTM on system-call sequences to catch exploits others miss","Unsupervised LSTM on Docker system calls detects web vulnerabilities with 0.97 AUC","Time-aware anomaly detection for web apps outperforms static-window methods"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000278,"raw_usage":{"total_tokens":1632,"prompt_tokens":899,"completion_tokens":733,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":515,"completion_tokens_details":{"reasoning_tokens":632}},"tokens_in":515,"tokens_out":733,"duration_ms":6708,"temperature":1.0,"reasoning_tokens":632,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T14:59:11.757040+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run AppMine's trained models on a day of real production traffic from the same applications, with routine housekeeping jobs such as backups, cron jobs, and log rotation labeled as legitimate, and measure the false positive rate; if those bursts of file and network system calls resemble the tested post-exploitation scripts, the reported AUC at fixed false-positive rates would fall sharply.","supporting_citations":[{"cited_title":"Process moni- toring on sequences of system call count vectors","cited_arxiv_id":null,"evidence_quote":"Supplies the system-call count vector representation used to build per-window features."},{"cited_title":"A sense of self for unix processes","cited_arxiv_id":null,"evidence_quote":"Foundational result that system-call sequences deviate under attack, motivating the whole approach."},{"cited_title":"Long short-term memory","cited_arxiv_id":null,"evidence_quote":"Defines the LSTM architecture that the model uses to capture temporal dependencies."},{"cited_title":"https://www.metasploit.com/","cited_arxiv_id":null,"evidence_quote":"Provides the exploit and post-exploitation modules used to generate attack data for evaluation."},{"cited_title":"https://www.seleniumhq.org/","cited_arxiv_id":null,"evidence_quote":"Supplies the browser automation behind the Interactor tool that generates legitimate user workloads."},{"cited_title":"https: //sysdig.com/opensource/","cited_arxiv_id":null,"evidence_quote":"The container monitoring agent that records the system-call traces feeding the machine-learning models."}],"review_version":1}