REVIEW 3 major objections 6 minor 45 references
AppMine: Behavioral Analytics for Web Application Vulnerability Detection
T0 review · 3 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read AppMine shows that an unsupervised LSTM, trained only on legitimate system-call sequences, detects web exploits that PCA and one-class SVM miss.
desk verdict A solid new testbed with unsupervised syscall anomaly detection, but the LSTM-vs-baseline comparison is confounded with context window and the headline improvement claim is overbroad. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is an LSTM sequence predictor over system-call frequency vectors. Each one-second window is represented as a vector counting how often each system call appears; the network takes a sequence of length 15, predicts the next vector, and the anomaly score is a distance between prediction and actual vector. The distance is weighted by each system call's inverse frequency in training, an adaptation of TF-IDF, which the paper shows separates the attack-distance distribution from the legitimate distribution far better than uniform weighting. This mechanism is what lets the model exploit dependencies over time rather than just per-window counts.
What would settle it
Run AppMine's trained models on a day of real production traffic from the same applications, with routine housekeeping jobs such as backups, cron jobs, and log rotation labeled as legitimate, and measure the false positive rate; if those bursts of file and network system calls resemble the tested post-exploitation scripts, the reported AUC at fixed false-positive rates would fall sharply.
Extended reading notes
Core claim
The paper's central discovery is that the temporal ordering of system-call frequency vectors carries most of the signal for distinguishing an exploited web application from a healthy one. Where PCA and OCSVM, which examine each one-second window of system-call counts in isolation, achieve AUCs of 0.81 and 0.83 on the Struts CVE-2017-5638 exploit, an LSTM that consumes the preceding 15 one-second frequency vectors and predicts the next vector reaches 0.97. The same pattern holds across most of the fifteen post-exploitation scripts on four applications, with LSTM average AUCs between 0.75 and 0.97. The authors argue this is because attack activity differs from legitimate use mainly in the transition structure between windows, not in any single window's histogram.
Load-bearing premise
The evaluation assumes that the synthetic legitimate workloads (browser sessions and FTP benchmark traffic) are representative enough of real user traffic that a model trained on them will transfer, and that normal production traffic never contains system call patterns similar to post-exploitation scripts.
Editorial extensions
If this is right
- An attacker who compromises a web application and runs the fifteen post-exploitation scripts tested produces a detectable deviation in the sequence of system-call counts even though individual windows may look normal.
- Because training uses only legitimate data, the same pipeline can be pointed at a new application or zero-day exploit without re-labeling attack data, as long as clean legitimate logs exist.
- A per-application threshold chosen on a legitimate validation set keeps false positives low at deployment time, and the LSTM's average AUC advantage over PCA and OCSVM on the testbed ranges from 0.09 to 0.25.
- The sequencing benefit is not uniform: on Drupal and one WordPress plugin, OCSVM posted slightly higher average AUC than LSTM, so the approach's advantage depends on the application.
Reading between the lines
- A natural extension the authors do not run is to test the same detector on legitimate bursts that mimic post-exploitation scripts, such as scheduled backups, package-manager runs, or admin commands, since those could produce false positives that the synthetic legitimate workload never exercises.
- The TF-IDF weighting suggests that much of the model's power may come from a few rare system calls; an ablation removing the rarest calls would reveal whether the detector is robust to call-set drift across kernel versions.
- If the sequencing signal transfers, a similar framework could monitor containerized databases or message queues, not just web front-ends, as long as a clean legitimate workload can be generated.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents AppMine, an unsupervised anomaly detection system for containerized web applications. The system collects system-call frequency vectors from Docker containers via Sysdig, then trains either PCA, one-class SVM, or an LSTM on legitimate data only. The LSTM consumes a sequence of 15 one-second frequency vectors and predicts the next vector, with an inverse-frequency-weighted distance used as the anomaly score. On a testbed with four web applications and seven recreated Metasploit exploits (15 post-exploitation scripts), the paper reports that the LSTM model outperforms the traditional baselines, with Struts CVE-2017-5638 average AUC of 0.97 versus 0.81 for PCA and 0.83 for OCSVM. The main claims are that unsupervised, sequence-aware modeling of system-call behavior can detect previously unseen exploit activity, and that the LSTM architecture provides a significant advantage over frequency-vector-only anomaly detection.
Significance. If the reported results hold, the paper would provide a useful data point that deep sequence models can detect exploit activity in containerized web applications without attack data for training. The testbed is carefully constructed, with separate legitimate and attack collection, and the evaluation uses standard AUC metrics. The TF-IDF-weighted prediction-error distance for the LSTM is a sensible, if simple, contribution. The paper also clearly states the threat model. However, the central claim of LSTM superiority is currently confounded by an input-representation mismatch and by hyperparameter selection on labeled test data, and the 'improves by 0.09 to 0.25' statement is contradicted by the paper's own Figure 7 for two of seven configurations. These issues are fixable but require additional experiments and textual qualification; the current version overstates the evidence.
major comments (3)
- [Section 3.4, Table 4] The comparison between the LSTM and the traditional baselines is confounded with input representation. PCA and OCSVM are said to take 'the system call frequency feature vector computed for one time interval' (one 1-second vector), while the LSTM receives a sequence of 15 such vectors (Table 4). Any AUC gain may therefore come from a longer context window rather than from the recurrent architecture's sequential modeling. The central claim that the LSTM 'exploit[s] the dependencies in sequences of system calls over time' would be supported by an ablation in which PCA/OCSVM receive the same 15-second context (e.g., a concatenated 15-vector input) or a feedforward network is trained on the same input. Without such a control, the attributed benefit of the LSTM is not isolated.
- [Section 4] The selection of the 1-second time window is made after comparing ROC curves for 100ms, 500ms, 1s, and 2s, with the text stating that the other options were worse because 'the ROC curves were closer to the diagonal.' ROC curves require the attack labels on the test set, so this is a form of test-set-based hyperparameter selection. It can inflate the reported AUCs for all models and could also influence the relative ranking. The authors should either select hyperparameters using only training/validation data (for example, by a criterion on legitimate data such as reconstruction error or prediction error on a held-out legitimate set) or report the AUCs for all models under each candidate window length, so that the sensitivity of the comparison is visible.
- [Figure 7 and Section 4, 'Comparison of LSTM with traditional models'] The statement that 'The AUC for LSTM is between 0.75 and 0.97 and improves the traditional models' average AUC between 0.09 and 0.25' is not supported by the paper's own aggregated results. In Figure 7(a) (Drupal), OCSVM achieves AUC 0.95 versus LSTM 0.93; in Figure 7(d) (WP Ajax Load More), OCSVM achieves AUC 0.86 versus LSTM 0.81. These are not small-gap outliers but direct counterexamples to the stated improvement range. The abstract and introduction should be qualified to reflect the configurations where LSTM outperforms the baselines, and the authors should discuss why the LSTM underperforms on those applications, for example in terms of workload characteristics or attack profiles.
minor comments (6)
- [Figure 6] The caption of Figure 6 says 'ROC curves for enum_system for Struts with CVE-2017-5638,' but the text in Section 4 refers to 'the attack script enum_network' and the plot legend reads 'enum_network.' The caption should be corrected to match the actual plotted script.
- [Section 3.4, LSTM training] The definition of the inverse-frequency weight f-bar_i in the weighted distance formula is not fully specified; the paper should state the exact formula used (for example, log(N/df_i) or a smoothed variant) and how it is computed from the training data.
- [Section 4, hyperparameters] The hyperparameters for OCSVM (kernel type, nu, gamma) are not reported, although PCA components and all LSTM settings are given. These parameters materially affect OCSVM's AUC and should be listed for reproducibility.
- [Section 4] The evaluation appears to be based on a single train/test split; reporting standard deviations or performing multiple random splits would strengthen the comparison and help assess whether the observed AUC differences are stable.
- [Section 5 and Section 6] There are minor typographical errors: Section 5 has 'Bayesian earning methods' instead of 'Bayesian learning methods,' and Section 6 has 'web application' where the plural is intended. The Swaddler description also uses 'worflow' instead of 'workflow.'
- [Section 3.1] The legitimate workloads are generated by Interactor (a Selenium-based simulator) and ftpbench; the paper should more explicitly discuss how the models might behave under noisier production workloads (e.g., variation in user concurrency or periodic administrative activity) and whether the reported false-positive rates would be expected to transfer.
Circularity Check
No circularity: LSTM predictions are evaluated on held-out attack traces with legitimate-only training and threshold selection.
full rationale
AppMine's derivation chain is not circular. The central claim is an empirical comparison: the LSTM predicts the next one-second system-call frequency vector from a 15-vector history, and its AUC is compared against PCA and OCSVM on held-out Metasploit attack traces. The LSTM is trained only on legitimate data ('AppMine leverages only legitimate application data for training a machine learning model that learns the system call distribution under normal conditions'), the anomaly threshold is set on a legitimate-only validation set ('we can select a threshold per application to minimize the False Positive rate during training'), and AUC is computed over test windows containing both legitimate and attack data. No equation in the paper defines the reported AUC as a function of the training targets or of any parameter fitted to attack labels. The TF-IDF inverse-frequency weights in the LSTM distance are computed from the legitimate training distribution, not from attack data, so the separation shown in Figure 5 is an empirical outcome rather than an identity. The only self-citation ([44], Beehive) appears in related work and is not load-bearing. The skeptic's concern that PCA and OCSVM receive one-second histograms while the LSTM receives fifteen one-second vectors is a genuine comparison confound, and the Section 4 statement that other window lengths were rejected because 'the ROC curves were closer to the diagonal' indicates hyperparameter selection using attack labels, which is a validity risk. However, neither issue makes the reported prediction equivalent to its inputs by construction. Under the strict standard requiring a specific reduction or a fitted parameter renamed as a prediction, there is no circular step.
Assumptions & free parameters
free parameters (4)
- Time window length =
1s
- PCA components =
20
- LSTM sequence size =
15
- LSTM hidden units =
100
assumptions (4)
- domain assumption Application exploits and cyber attacks result in observable deviations in system call frequency distributions compared to normal operation.
- domain assumption The synthetic legitimate workloads (Interactor and ftpbench) are representative of real user behavior for the web applications.
- domain assumption Containers and monitoring agents are trusted and not under attacker control; the adversary interacts only remotely before exploitation.
- standard math Standard properties of PCA, one-class SVM, and LSTM training hold as described in the cited literature.
Cite this review
Pith. "Pith review of AppMine: Behavioral Analytics for Web Application Vulnerability Detection." pith.science (2026). https://pith.science/paper/BXAXWPPP
@misc{pith2026190801928,
author = {Pith},
title = {Pith review of: AppMine: Behavioral Analytics for Web Application Vulnerability Detection},
year = {2026},
howpublished = {\url{https://pith.science/paper/BXAXWPPP}},
note = {Machine review of arXiv:1908.01928}
}
read the original abstract
Web applications in widespread use have always been the target of large-scale attacks, leading to massive disruption of services and financial loss, as in the Equifax data breach. It has become common practice to deploy web application in containers like Docker for better portability and ease of deployment. We design a system called AppMine for lightweight monitoring of web applications running in Docker containers and detection of unknown web vulnerabilities. AppMine is an unsupervised learning system, trained only on legitimate workloads of web application, to detect anomalies based on either traditional models (PCA and one-class SVM), or more advanced neural-network architectures (LSTM). In our evaluation, we demonstrate that the neural network model outperforms more traditional methods on a range of web applications and recreated exploits. For instance, AppMine achieves average AUC scores as high as 0.97 for the Apache Struts application (with the CVE-2017-5638 exploit used in the Equifax breach), while the AUC scores for PCA and one-class SVM are 0.81 and 0.83, respectively.
Figures
Figures from the paper (6 more)
Reference graph
Works this paper leans on
-
[1]
Zubair Shafiq, and Muddassar Fa- rooq
Faraz Ahmed, Haider Hameed, M. Zubair Shafiq, and Muddassar Fa- rooq. Using spatio-temporal information in api calls with machine learning algorithms for malware detection. In Proceedings of the 2nd ACM Workshop on Security and Artificial Intelligence , AISec ’09, pages 55–62, New York, NY, USA, 2009. ACM
work page 2009
-
[2]
Saner: Com- posing static and dynamic analysis to validate sanitization in web applications
Davide Balzarotti, Marco Cova, Vika Felmetsger, Nenad Jovanovic, Engin Kirda, Christopher Kruegel, and Giovanni Vigna. Saner: Com- posing static and dynamic analysis to validate sanitization in web applications. In Proceedings of the IEEE Symposium on Security and Privacy, 2008
work page 2008
-
[3]
Christopher M. Bishop. Pattern recognition and machine learning, 5th Edition. Information science and statistics. Springer, 2007
2007
-
[4]
Swaddler: An approach for the anomaly-based detection of state violations in web applications
Marco Cova, Davide Balzarotti, Viktoria Felmetsger, and Giovanni Vigna. Swaddler: An approach for the anomaly-based detection of state violations in web applications. In Christopher Kruegel, Richard Lippmann, and Andrew Clark, editors, Recent Advances in Intrusion Detection, pages 63–86, Berlin, Heidelberg, 2007. Springer Berlin Hei- delberg
work page 2007
- [5]
- [6]
-
[7]
Deeplog: Anomaly detection and diagnosis from system logs through deep learning
Min Du, Feifei Li, Guineng Zheng, and Vivek Srikumar. Deeplog: Anomaly detection and diagnosis from system logs through deep learning. In Proceedings of the 2017 ACM SIGSAC Conference on Com- puter and Communications Security , CCS ’17, pages 1285–1298, New York, NY, USA, 2017. ACM
work page 2017
-
[8]
Process moni- toring on sequences of system call count vectors
Michael Dymshits, Benjamin Myara, and David Tolpin. Process moni- toring on sequences of system call count vectors. InSecurity Technology (ICCST), 2017 International Carnahan Conference on , pages 1–5. IEEE, 2017
work page 2017
Show all 45 references
-
[9]
https: //www.equifaxsecurity2017.com/, 2017
Cybersecurity incident and important consumer information. https: //www.equifaxsecurity2017.com/, 2017
2017
-
[10]
A sense of self for unix processes
Stephanie Forrest, Steven A Hofmeyr, Anil Somayaji, and Thomas A Longstaff. A sense of self for unix processes. In Security and Privacy,
-
[11]
https://github.com/selectel/ftpbench
ftpbench. https://github.com/selectel/ftpbench
-
[12]
Long short-term memory
Sepp Hochreiter and Jurgen Schmidhuber. Long short-term memory. Neural Computation, 9:1735âĂŞ–1780, 1997
1997
-
[13]
Intrusion detection using sequences of system calls
Steven A Hofmeyr, Stephanie Forrest, and Anil Somayaji. Intrusion detection using sequences of system calls. Journal of computer security, 6(3):151–180, 1998
1998
-
[14]
Pixy: A static analysis tool for detecting web application vulnerabilities (short paper)
Nenad Jovanovic, Christopher Kruegel, and Engin Kirda. Pixy: A static analysis tool for detecting web application vulnerabilities (short paper). In Proceedings of the 2006 IEEE Symposium on Security and Privacy , SP ’06, pages 258–263, Washington, DC, USA, 2006. IEEE Computer Society
2006
-
[15]
Bayesian event classification for intrusion detection
Christopher Kruegel, Darren Mutz, William Robertson, and Fredrik Valeur. Bayesian event classification for intrusion detection. In Pro- ceedings of the 29th Annual Computer Security Applications Conference , ACSAC ’03, 2003
2003
-
[16]
A multi- model approach to the detection of web-based attacks
Christopher Kruegel, Giovanni Vigna, and William Robertson. A multi- model approach to the detection of web-based attacks. Comput. Netw., 48(5):717–738, August 2005
2005
-
[17]
An application of machine learning to anomaly detection
Terran Lane and Carla E Brodley. An application of machine learning to anomaly detection. In Proceedings of the 20th National Information Systems Security Conference , volume 377, pages 366–380. Baltimore, USA, 1997
1997
-
[18]
Sequence matching and learning in anomaly detection for computer security
Terran Lane, Carla E Brodley, et al. Sequence matching and learning in anomaly detection for computer security. In AAAI Workshop: AI Approaches to Fraud Detection and Risk Management , pages 43–49. 11 Providence, Rhode Island, 1997
1997
-
[19]
Data mining approaches for intrusion detection
Wenke Lee, Salvatore J Stolfo, et al. Data mining approaches for intrusion detection. In USENIX Security Symposium, pages 79–93. San Antonio, TX, 1998
1998
-
[20]
Lukasz Lenart. S2-052. https://cwiki.apache.org/confluence/display/ WW/S2-052, 2017
2017
-
[21]
Detecting intrusions through system call sequence and argument analysis
Federico Maggi, Matteo Matteucci, and Stefano Zanero. Detecting intrusions through system call sequence and argument analysis. IEEE Trans. Dependable Secur. Comput., 7(4):381–395, October 2010
2010
-
[22]
Medeiros, N
I. Medeiros, N. Neves, and M. Correia. Detecting and removing web application vulnerabilities with static analysis and data mining. IEEE Transactions on Reliability, 65(1):54–69, March 2016
2016
-
[23]
https://www.metasploit.com/
Metasploit. https://www.metasploit.com/
-
[24]
Exploiting execution context for the detection of anomalous system calls
Darren Mutz, William Robertson, Giovanni Vigna, and Richard Kem- merer. Exploiting execution context for the detection of anomalous system calls. In Christopher Kruegel, Richard Lippmann, and An- drew Clark, editors, Recent Advances in Intrusion Detection, pages 1–20, Berlin, ...
2007
-
[25]
Anomalous system call detection
Darren Mutz, Fredrik Valeur, Giovanni Vigna, and Christopher Kruegel. Anomalous system call detection. ACM Transactions on Information and System Security (TISSEC) , 9(1):61–93, 2006
2006
-
[26]
Peisert, M
S. Peisert, M. Bishop, S. Karin, and K. Marzullo. Analysis of computer intrusions using sequences of function calls. Dependable and Secure Computing, IEEE Transactions on , 4(2):137 –150, april-june 2007
2007
-
[27]
http://www.proftpd.org/
The ProFTPD Project. http://www.proftpd.org/
-
[28]
Effec- tive anomaly detection with scarce training data
William Robertson, Christopher Kruegel, and Giovanni Vigna. Effec- tive anomaly detection with scarce training data. In Proc. Network and Distributed System Security Symp. (NDSS) , 2010
2010
-
[29]
Kemmerer
William Robertson, Giovanni Vigna, Christopher Kruegel, and Richard A. Kemmerer. Using generalization and characterization tech- niques in the anomaly-based detection of web attacks. InProc. Network and Distributed System Security Symp. (NDSS) , 2010
2010
-
[30]
Rumelhart, G
D. Rumelhart, G. Hinton, and R. Williams. Learning representations by back-propagating errors. Nature, 323:533âĂŞ–536, 1986
1986
-
[31]
Leveraging user interactions for in-depth testing of web applications
Allister S., Kirda E., and Kruegel C. Leveraging user interactions for in-depth testing of web applications. In Proceedings of Recent Advances in Intrusion Detection, RAID, 2008
2008
-
[32]
Preventing input validation vulnerabilities in web applications through automated type analysis
Theodoor Scholte, William Robertson, Davide Balzarotti, and Engin Kirda. Preventing input validation vulnerabilities in web applications through automated type analysis. In Proceedings of the 2012 IEEE 36th Annual Computer Software and Applications Conference, COMPSAC ’12, pag...
2012
-
[33]
https://www.seleniumhq.org/
Selenium - Web Browser Automation. https://www.seleniumhq.org/
-
[34]
L. K. Shar, L. C. Briand, and H. B. K. Tan. Web application vulnerability prediction using hybrid program analysis and machine learning. IEEE Transactions on Dependable and Secure Computing , 12(6):688–707, Nov 2015
2015
-
[35]
Outside the closed world: On us- ing machine learning for network intrusion detection
Robin Sommer and Vern Paxson. Outside the closed world: On us- ing machine learning for network intrusion detection. In 2010 IEEE symposium on security and privacy , pages 305–316. IEEE, 2010
2010
-
[36]
Spectrogram: A mixture-of-markov-chains model for anomaly detection in web traffic
Yingbo Song, Angelos D Keromytis, and Salvatore Stolfo. Spectrogram: A mixture-of-markov-chains model for anomaly detection in web traffic. In Proc. Network and Distributed System Security Symp. (NDSS) , 2009
2009
-
[37]
https: //sysdig.com/opensource/
Sysdig: Open Source Container Troubleshooting & Forensics. https: //sysdig.com/opensource/
-
[38]
Cross-site scripting prevention with dynamic data tainting and static analysis
Philipp Vogt, Florian Nentwich, Nenad Jovanovic, Engin Kirda, Christo- pher Kruegel, and Giovanni Vigna. Cross-site scripting prevention with dynamic data tainting and static analysis. In Proceedings of the 14th Annual Network & Distributed System Security Symposium , NDSS, 2007
2007
-
[39]
De- tecting intrusions using system calls: Alternative data models
Christina Warrender, Stephanie Forrest, and Barak Pearlmutter. De- tecting intrusions using system calls: Alternative data models. In Proceedings of the 1999 IEEE symposium on security and privacy (Cat. No. 99CB36344), pages 133–145. IEEE, 1999
1999
-
[40]
https:// wordpress.org/
Blog Tool, Publishing Platform, and CMS - WordPress. https:// wordpress.org/
-
[41]
https://www.exploit-db.com/exploits/38660
wpajaxloadmore. https://www.exploit-db.com/exploits/38660
-
[42]
https://www.exploit-db.com/exploits/36810
wpnmedia. https://www.exploit-db.com/exploits/36810
-
[43]
https://www.exploit-db.com/exploits/36809
wpreflexgallery. https://www.exploit-db.com/exploits/36809
-
[44]
Robertson, Ari Juels, and Engin Kirda
Ting-Fang Yen, Alina Oprea, Kaan Onarlioglu, Todd Leetham, William K. Robertson, Ari Juels, and Engin Kirda. Beehive: Large-scale log analysis for detecting suspicious activity in enterprise networks. In Annual Computer Security Applications Conference, ACSAC ’13, New Orleans,...
2013
-
[1996]
IEEE, 1996
Proceedings., 1996 IEEE Symposium on , pages 120–128. IEEE, 1996
1996
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.