REVIEW 5 major objections 5 minor 36 references
User-Based Sequential Modeling with Transformer Encoders for Insider Threat Detection
T0 review · 5 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read A transformer trained only on normal user activity, fed per-user session sequences, flags 99.43% of simulated insider attacks on a combined CERT test set.
desk verdict The UBS idea is sensible, but the evaluation leaks training users into the test set, so the headline numbers do not support the state-of-the-art claim. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is User-Based Sequencing (UBS): a rearrangement of raw CERT log features into a per-user tensor of shape [501 days × 9 sessions per day × 35 features], so that one user's whole history is one multi-index sequence. A transformer encoder with an initial linear embedding, positional encoding, six encoder blocks, and eight attention heads is trained with batch size one (one user at a time) to reconstruct benign input; mean squared error between input and output defines the reconstruction error. That scalar error per user/session is then handed to One-Class SVM, Local Outlier Factor, or Isolation Forest, each of which decides whether the error pattern is normal or anomalous. The machinery works by turning 'is this user behaving normally?' into 'can the model rebuild this user's behavior from patterns learned on benign users?'.
What would settle it
Fit OCSVM, LOF, and iForest on reconstruction errors from the benign r4.2 training users only, freeze those detectors, then score the r5.2 and r6.2 users in Test-4 and report per-release recall and FPR; if recall drops materially below 99% or FPR exceeds the reported 0.0571, the headline figures depend on test-set information rather than on the learned notion of normal behavior.
Extended reading notes
Core claim
The paper's central claim is that transforming structured log data into per-user temporal sequences, then learning to reconstruct those sequences with a transformer encoder trained exclusively on benign activity, yields an insider-threat detector that outperforms both tabular machine learning and conventional autoencoders. On Test-4, a combined test set of 210 benign and 174 malicious users drawn from CERT r4.2, r5.2, and r6.2, the paper reports 96.61% accuracy, 99.43% recall, 96.38% F1-score, and 95.00% AUROC for the transformer pipeline, with false-negative rate 0.0057 and false-positive rate 0.0571. The paper further reports that the same UBS reorganization lifts the autoencoder baseline substantially, which it takes as evidence that the sequencing structure, not the transformer alone, carries much of the benefit.
Load-bearing premise
The model learns normality only from benign users in CERT r4.2, yet its main test set includes users from CERT r5.2 and r6.2, so the result depends on the assumption that benign behavior, and the scale of the resulting reconstruction errors, is comparable across CERT releases.
Editorial extensions
If this is right
- Presenting CERT logs as per-user temporal sequences improves both the transformer and an autoencoder baseline, so UBS is a reusable data transformation, not a transformer-only trick.
- The pipeline is geared to the insider-threat setting where missed attacks are the costliest failure: the reported recall is near-perfect and the false-negative rate is the lowest among the baselines the paper tabulates.
- A model trained on benign r4.2 users is reported to detect malicious users drawn from r5.2 and r6.2, indicating cross-release generalization on synthetic CERT data.
- Because the three outlier detectors give similar results on the transformer's reconstruction errors, the errors themselves appear to be a strong anomaly signal regardless of the final classifier.
- Session-level granularity is meant to catch behavioral shifts that day- or week-level aggregation would smooth away.
Reading between the lines
- The reported Test-4 metrics assume a test population that is roughly 45% malicious; real deployments have far lower base rates, so thresholds and scores would need recalibration before the accuracy and F1 numbers transfer.
- The paper does not isolate the contribution of temporal order from the contribution of per-user distributions; a shuffled-session control would show how much of the gain actually comes from sequence structure.
- The autoencoder improvement under UBS suggests a broader recipe: reorganize any structured log into per-user sequences, train any reconstruction model on benign data, and score reconstruction errors with off-the-shelf outlier detectors; this is testable on other behavioral log domains, such as network flows or physical access records.
- Reporting per-release results for Test-4 would show which CERT release drives the combined scores; Test-3, with only five malicious users, is the least informative of the four test sets.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a User-Based Sequencing (UBS) methodology that converts raw CERT insider-threat logs into per-user tensors of shape [days, sessions, features], then trains a Transformer encoder on benign data to reconstruct these tensors. Reconstruction errors are scored by OCSVM, LOF, and Isolation Forest. The authors report four test sets, including a mixed-release Test-4, and claim state-of-the-art detection performance (96.61% accuracy, 99.43% recall, 96.38% F1, 95% AUROC) and very low false negative and false positive rates.
Significance. If the reported numbers were reproducible and the evaluation were clean, the paper would make a credible contribution to insider-threat detection by demonstrating that sequential modeling with Transformers can outperform tabular and simple autoencoder baselines, and the UBS representation could be reused by other sequence models. The paper is explicit about the architecture, hyperparameters, and the unsupervised nature of the anomaly scorers. However, the lack of a proper train/test split, the unspecified training of the outlier detectors, the absence of error bars, and the uncontrolled comparison with prior work currently preclude accepting the state-of-the-art claim.
major comments (5)
- [3.6.3 / Table 1 vs. 3.3–3.4] The model is trained on benign users from CERT r4.2, but Test-1 includes 30 benign r4.2 users and Test-4 combines Tests 1–3, so those same users are part of the evaluation. The paper never states that these 30 users were excluded from training. The only separation claim (Section 5) is that the test sets do not overlap with each other; it does not establish train/test disjointness. If the benign r4.2 users in Test-1/Test-4 were in the training set, the Transformer can memorize their sequences and produce near-zero reconstruction errors, making anomaly detection trivially easier. This directly affects the headline numbers in Table 3 and the abstract. The authors must either exclude all training users from the test sets or demonstrate performance on a genuinely held-out set of r4.2 benign users.
- [3.6.1] The paper does not specify how OCSVM, LOF, and iForest are trained and thresholded. It is essential to know whether these detectors are fit on reconstruction errors of benign training users (a legitimate novelty-detection setup) or on reconstruction errors of the test set (which would leak label information and make the reported metrics optimistic). The hyperparameters for these detectors (e.g., nu/gamma for OCSVM, n_neighbors for LOF, contamination for iForest) are not reported, so the protocol is not reproducible. This is load-bearing because the same reconstruction errors can yield dramatically different F-scores depending on the detector configuration.
- [3.6] The hyperparameter search is described as a Cartesian product, but no validation split is mentioned. Selecting hyperparameters by optimizing performance on the test sets would inflate the reported results. The authors should describe a validation procedure using only benign data (e.g., a held-out subset of r4.2 benign users) and report the selected hyperparameters.
- [5.1 / Table 7] The comparison with prior work is not on equal footing. The baseline models are evaluated on single CERT releases (mostly r4.2), while the proposed model is evaluated on the combined Test-4 set. Differences in test composition, contamination ratio, and preprocessing make the 'state-of-the-art' claim unsupported. The authors should compare on the same evaluation protocol or explicitly quantify the effect of the different test sets.
- [Tables 2 and 3] Results are reported for a single run; no standard deviations or multiple seeds are provided. Given the small number of malicious users in some test sets (e.g., 5 in Test-3), the reported metrics are subject to considerable variance. The authors should report mean and variance over multiple runs, or at least specify the random seeds.
minor comments (5)
- [5 (Tables 4–6)] The analysis of false positive rate is incorrect. For example, Table 4 says 'FPR + 76.01% Reduction in the rate at which true positives are mistakenly overlooked,' but FPR is the rate of false positives among negatives, not a measure of missed true positives (which is FNR). Table 5 similarly labels a 4.28-percentage-point increase in FPR as 'the rate at which true positives are mistakenly overlooked.' These statements need to be corrected.
- [3.6] The citation [24] is to the Cartesian product algorithm for type inference, not to hyperparameter search; a different reference is needed.
- [3.5] The notation 'user_data: U -> T' is ambiguous; it should clearly define that U is the set of users and T is the space of tensors of shape [501,9,35], with a formal mapping from each user to their tensor.
- [Table 7] The notation '+1', '+2', '+3', '+4' in the baseline names is undefined; these should be replaced with meaningful names or a footnote explaining the notation.
- [Abstract] Calling a 5.71% false positive rate 'exceptionally low' is a matter of interpretation, but it is not a technical error.
Circularity Check
Test-1 and part of Test-4 evaluate the Transformer on benign r4.2 users that the paper's training description does not exclude from training, so the headline metrics include fitted reconstruction values rather than fresh predictions.
-
fitted input called prediction
[Sections 3.3, 3.4, 3.6.3, Table 1, Section 5]
"In our research, we used version r4.2 to train our model (Sec. 3.3); we use our novel UBS structure to train the Transformer model on only benign data (Sec. 3.4); Table 1 lists Test-1 as 'r4.2 | 30 | 70 | 100'; Sec. 5 says 'Every test we have designed is based on a stratified splitting approach to ensure complete data separation and that the data in each set are unique; no overlap is present in any of the sets.'"
The training description uses the benign portion of r4.2 with no holdout split, and Test-1 then scores 30 benign r4.2 users from that same release. Test-4 inherits those 30 users (its 210 benign total equals 30+60+120). The only separation sentence guarantees uniqueness of data within/across test sets; it never excludes these r4.2 benign users from training. Consequently, for these users the 'reconstruction errors' are the MSE values the Transformer was explicitly optimized to minimize, so classifying them as normal is the training objective, not an anomaly-detection prediction. This forces Test-1's perfect recall/FNR 0.0 and inflates Test-4's 96.61% accuracy / 99.43% recall / 0.0057 FNR. Test-2/3 are from other releases and provide some independent support.
full rationale
The core derivation (train a Transformer encoder on benign activity, use reconstruction error as an anomaly score, then feed that score to OCSVM/LOF/iForest) is internally coherent and is not, by itself, circular. The main circularity-adjacent problem is evaluation leakage: the paper says it used r4.2 to train on 'only benign data,' but Table 1 places 30 benign r4.2 users into Test-1, and Test-4 is the arithmetic union of Tests 1-3 (30+60+120 benign users, 70+99+5 malicious users). The Section 5 statement about 'complete data separation' only claims that the test sets do not overlap each other; it does not assert disjointness from the training set. As written, the benign r4.2 users in Test-1 and Test-4 may be training users, in which case their low reconstruction errors are the very MSE values the model was fit to produce, making their 'detection' as benign statistically forced. This partially explains the perfect recall and zero FNR on Test-1 and contributes to the headline Test-4 numbers. There is independent content elsewhere: Test-2 (r5.2, 99 malicious users) and Test-3 (r6.2) are not contaminated by r4.2 training, and the Transformer still reports strong recall and AUC there, so the central idea is not wholly reduced to a fit. The paper also never specifies whether the outlier detectors are fit on training reconstruction errors or test reconstruction errors; if the latter, that would be additional leakage, but because the text does not state this, I do not count it as a separate circular step. The post-hoc selection of the best detector and metric for the headline (iForest for accuracy/recall/FPR, OCSVM for recall) is a reporting concern rather than a circular derivation. No self-citations are load-bearing here. Overall, the evaluation is partially circular as written, but the method itself retains independent empirical content, so the score is moderate rather than extreme.
Assumptions & free parameters
free parameters (4)
- Transformer hyperparameters (6 encoder layers, 8 heads, dropout 0.1, learning rate 1e-5) =
5,387,579 trainable parameters
- Max sessions per day S =
9
- Feature count F =
35
- Outlier detector hyperparameters (OCSVM nu/gamma, LOF neighbors, iForest contamination) =
not reported
assumptions (4)
- domain assumption The filtered r4.2 training subset contains only benign user activity.
- domain assumption Reconstruction error from a benign-trained transformer is a valid anomaly score.
- domain assumption CERT synthetic logs are representative of real insider threat behavior.
- domain assumption The per-user reconstruction error vectors can be compared across users and fed to OCSVM, LOF, and iForest.
Cite this review
Pith. "Pith review of User-Based Sequential Modeling with Transformer Encoders for Insider Threat Detection." pith.science (2026). https://pith.science/paper/YBRTFUQ3
@misc{pith2026250623446,
author = {Pith},
title = {Pith review of: User-Based Sequential Modeling with Transformer Encoders for Insider Threat Detection},
year = {2026},
howpublished = {\url{https://pith.science/paper/YBRTFUQ3}},
note = {Machine review of arXiv:2506.23446}
}
read the original abstract
Insider threat detection presents unique challenges due to the authorized status of malicious actors and the subtlety of anomalous behaviors. Existing machine learning methods often treat user activity as isolated events, thereby failing to leverage sequential dependencies in user behavior. In this study, we propose a User-Based Sequencing (UBS) methodology, transforming the CERT insider threat dataset into structured temporal sequences suitable for deep sequential modeling. We deploy a Transformer Encoder architecture to model benign user activity and employ its reconstruction errors as anomaly scores. These scores are subsequently evaluated using three unsupervised outlier detection algorithms: One-Class SVM (OCSVM), Local Outlier Factor (LOF), and Isolation Forest (iForest). Across four rigorously designed test sets, including combinations of multiple CERT dataset releases, our UBS-Transformer pipeline consistently achieves state-of-the-art performance - notably 96.61% accuracy, 99.43% recall, 96.38% F1-score, 95.00% AUROC, and exceptionally low false negative (0.0057) and false positive (0.0571) rates. Comparative analyses demonstrate that our approach substantially outperforms tabular and conventional autoencoder baselines, underscoring the efficacy of sequential user modeling and advanced anomaly detection in the insider threat domain.
Figures
Reference graph
Works this paper leans on
- [1]
-
[2]
Defining insider threats, 2023
CISA - Defining Insider Threats. Defining insider threats, 2023
work page 2023
-
[3]
Understanding controls to detect and mitigate malicious privileged user abuse
Harris Clyde Devince. Understanding controls to detect and mitigate malicious privileged user abuse. 4 2020
work page 2020
-
[4]
Common sense guide to mitigating insider threats, seventh edition
Software Engineering Institute. Common sense guide to mitigating insider threats, seventh edition. Technical report, Sep 2022
work page 2022
-
[5]
Deep learning for insider threat detection: Review, challenges and opportunities
Shuhan Yuan and Xintao Wu. Deep learning for insider threat detection: Review, challenges and opportunities. Computers and Security, 104, 2021
work page 2021
-
[6]
Michele Maasberg, Xiao Zhang, Myung Ko, Stewart R Miller, and Nicole Lang Beebe. An analysis of motive and observable behavioral indicators associated with insider cyber-sabotage and other attacks. IEEE Engineering Management Review, 48:151–165, 2020
work page 2020
-
[7]
Bridging the gap: A pragmatic approach to generating insider threat data
Joshua Glasser and Brian Lindauer. Bridging the gap: A pragmatic approach to generating insider threat data. pages 98 – 104, San Francisco, CA, United states, 2013
work page 2013
-
[8]
Attention is all you need
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017
2017
Show all 36 references
-
[9]
Mohan, and Ambairam Muthu Sivakrishna
Krunal Randive, R. Mohan, and Ambairam Muthu Sivakrishna. An efficient pattern-based approach for insider threat classification using the image-based feature representation.Journal of Information Security and Applications, 73, 2023
2023
-
[10]
Relational deep learning detection with multi-sequence representation for insider threats
Abdullah Alshehri. Relational deep learning detection with multi-sequence representation for insider threats. International Journal of Advanced Computer Science and Applications , 13:758–765, 2022
2022
-
[11]
Salman, Mariam M
Muhanned AlSlaiman, Mohammed I. Salman, Mariam M. Saleh, and Bin Wang. Enhancing false negative and positive rates for efficient insider threat detection. Computers and Security, 126, 2023
2023
-
[12]
A graph empowered insider threat detection framework based on daily activities
Wei Hong, Jiao Yin, Mingshan You, Hua Wang, Jinli Cao, Jianxin Li, Ming Liu, and Chengyuan Man. A graph empowered insider threat detection framework based on daily activities. ISA transactions, 141:84–92, 2023
2023
-
[13]
Hunting for insider threats using lstm-based anomaly detection
Miguel Villarreal-Vasquez, Gaspar Modelo-Howard, Simant Dube, and Bharat Bhargava. Hunting for insider threats using lstm-based anomaly detection. IEEE Transactions on Dependable and Secure Computing , 20(1):451 – 462, 2023
2023
-
[14]
Temporal feature aggregation with attention for insider threat detection from activity logs
Preetam Pal, Pratik Chattopadhyay, and Mayank Swarnkar. Temporal feature aggregation with attention for insider threat detection from activity logs. Expert Systems with Applications , 224:119925, 8 2023
2023
-
[15]
Deep transfer learning & beyond: Transformer language models in information systems research
Ross Gruetzemacher and David Paradice. Deep transfer learning & beyond: Transformer language models in information systems research. ACM Computing Surveys (CSUR), 54(10s):1–35, 2022. 10 A PREPRINT - AUGUST 20, 2025
2022
-
[16]
Fundamentals of recurrent neural network (rnn) and long short-term memory (lstm) network
Alex Sherstinsky. Fundamentals of recurrent neural network (rnn) and long short-term memory (lstm) network. Physica D: Nonlinear Phenomena , 404:132306, 2020
2020
-
[17]
Word embedding attention and balanced cross entropy technique for sentiment analysis
Vijaya Ravindra Sagvekar and Prashant Sharma. Word embedding attention and balanced cross entropy technique for sentiment analysis. Multiagent and Grid Systems , 19(1):23 – 42, 2023
2023
-
[18]
Pytorch: An imperative style, high-performance deep learning library
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems , 32, 2019
2019
-
[19]
Insider threat test dataset, 2020
Brian Lindauer. Insider threat test dataset, 2020
2020
-
[20]
Dtitd: An intelligent insider threat detection framework based on digital twin and self-attention based deep learning models
Wang and A El Saddik. Dtitd: An intelligent insider threat detection framework based on digital twin and self-attention based deep learning models. IEEE Access, 11:114013 – 114030, 2023
2023
-
[21]
Generating long sequences with sparse transformers
Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509, 2019
1904 arXiv
-
[22]
Anovit: Unsupervised anomaly detection and localization with vision transformer-based encoder-decoder
Yunseung Lee and Pilsung Kang. Anovit: Unsupervised anomaly detection and localization with vision transformer-based encoder-decoder. IEEE Access, 10:46717 – 46724, 2022. Anomaly detection;Anomaly localiza- tions;Features extraction;Images reconstruction;Location awareness;MVT...
2022
-
[23]
Adam: A method for stochastic optimization
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014
2014 arXiv
-
[24]
The cartesian product algorithm: Simple and precise type inference of parametric polymorphism
Ole Agesen. The cartesian product algorithm: Simple and precise type inference of parametric polymorphism. In ECOOP’95—Object-Oriented Programming, 9th European Conference, Åarhus, Denmark, August 7–11, 1995 9 , pages 2–26. Springer, 1995
1995
-
[25]
Hyperspectral anomaly detection via memory-augmented autoencoders
Zhe Zhao and Bangyong Sun. Hyperspectral anomaly detection via memory-augmented autoencoders. CAAI Transactions on Intelligence Technology, 8(4):1274 – 1287, 2023. Abnormal samples;Auto encoders;Background reconstruction;Hyperspectral anomaly detection;Hyperspectral Data;Low d...
2023
-
[26]
Adaptive one-class ensemble-based anomaly detection: An application to insider threats
Diana Haidar and Mohamed Medhat Gaber. Adaptive one-class ensemble-based anomaly detection: An application to insider threats. volume 2018-July, Rio de Janeiro, Brazil, 2018
2018
-
[27]
A meta-analysis of the anomaly detection problem
Andrew Emmott, Shubhomoy Das, Thomas Dietterich, Alan Fern, and Weng-Keen Wong. A meta-analysis of the anomaly detection problem. arXiv preprint arXiv:1503.01158, 2015
2015 arXiv
-
[28]
Machine learning and anomaly detection for insider threat detection
Filip Wieslaw Bartoszewski. Machine learning and anomaly detection for insider threat detection . PhD thesis, Heriot-Watt University, 2022
2022
-
[29]
Translog: A unified transformer-based framework for log anomaly detection
Hongcheng Guo, Xingyu Lin, Jian Yang, Yi Zhuang, Jiaqi Bai, Bo Zhang, Tieqiao Zheng, and Zhoujun Li. Translog: A unified transformer-based framework for log anomaly detection. 2021
2021
-
[30]
Insider threat prediction based on unsupervised anomaly detection scheme for proactive forensic investigation
Yichen Wei, Kam-Pui Chow, and Siu-Ming Yiu. Insider threat prediction based on unsupervised anomaly detection scheme for proactive forensic investigation. F orensic Science International: Digital Investigation, 38, 2021
2021
-
[31]
Anju and M
A. Anju and M. Krishnamurthy. M-eos: modified-equilibrium optimization-based stacked cnn for insider threat detection. Wireless Networks, 30(4):2819 – 2838, 2024
2024
-
[32]
Lan: Learning adaptive neighbors for real-time insider threat detection
Xiangrui Cai, Yang Wang, Sihan Xu, Hao Li, Ying Zhang, and Xiaojie Yuan. Lan: Learning adaptive neighbors for real-time insider threat detection. arXiv preprint arXiv:2403.09209, 2024
2024 arXiv
-
[33]
Itdbert: Temporal-semantic representa- tion for insider threat detection
Weiqing Huang, He Zhu, Ce Li, Qiujian Lv, Yan Wang, and Haitian Yang. Itdbert: Temporal-semantic representa- tion for insider threat detection. In 2021 IEEE Symposium on Computers and Communications (ISCC) , pages 1–7. IEEE, 2021
2021
-
[34]
Behavioral based insider threat detection using deep learning
Rida Nasir, Mehreen Afzal, Rabia Latif, and Waseem Iqbal. Behavioral based insider threat detection using deep learning. IEEE Access, 9:143266–143274, 2021
2021
-
[35]
Britd: behavior rhythm insider threat detection with time awareness and user adaptation
Shuang Song, Neng Gao, Yifei Zhang, and Cunqing Ma. Britd: behavior rhythm insider threat detection with time awareness and user adaptation. Cybersecurity, 7(1), 2024
2024
-
[36]
Zainal Abidin, and S.N
Al-Mhiqani, Rabiah Ahmed, Z.A. Zainal Abidin, and S.N. Isnin. An integrated imbalanced learning and deep neural network model for insider threat detection. International Journal of Advanced Computer Science and Applications, 12(1):573 – 577, 2021. 11
2021
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.