REVIEW 4 major objections 5 minor 30 references
Take Package as Language: Anomaly Detection Using Transformer
T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read Packet-as-language GPT reports 1.00 macro-F1 on rare attacks.
desk verdict The digit-level tokenizer is a real idea; the evaluation's unstated split and mislabeled metric make the headline 100% F1 unsupported. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing machinery is a deliberately unusual tokenizer plus a three-part embedding. The tokenizer reverses the character order of every numeric field, converts each digit into its own token, inserts separators $S_i = S_0 + i - 1$ that both mark field boundaries and encode field order, and appends the class label as the final integer token. Each token then receives the sum of three embeddings: a word embedding for its digit value, a numeric-position embedding indicating whether the digit is in the ones, tens, or hundreds place of its field, and a field-position embedding indicating which packet field it came from. A GPT-2-style causal Transformer decoder is trained with next-token negative log-likelihood over all tokens, and at inference the model reads out the label by taking the argmax of the probability at the label position. This design is what lets the model treat classification as language modelling and, the paper argues, is why it survives extreme imbalance.
What would settle it
A concrete check: re-run the CICIDS2017 experiment with a temporal split that trains on flows captured on the first days and tests on flows captured on later days, or group flows by attack session or timestamp so no session spans the split, and report macro-F1 at the same three imbalance ratios. If macro-F1 drops substantially below 1.00 under that split, the perfect scores are an artifact of train/test similarity rather than a property of the packet-as-language objective.
Extended reading notes
Core claim
On the paper's own terms, the central discovery is that a causal language model trained to predict the next token of a packet, where each packet is serialized digit-by-digit with reversed numeric fields and the class label placed as the last token, learns to classify network traffic so well that it attains 1.00 macro Precision, Recall, and F1 on CICIDS2017 at imbalance ratios of 0.001, 0.0005, and 0.0002, outperforming Extra Trees, Random Forest, an optimized random forest, and a CNN-BiLSTM baseline. In a one-shot protocol averaged over ten random draws, it reports 0.91 precision, 0.82 recall, and 0.84 macro-F1 with one training sample per class. The same model, retrained with the same hyperparameters on the car-hacking CAN bus dataset, reports 1.00 on all three metrics, matching a transfer-CNN baseline. Attention-weight visualizations show the model attends to inter-arrival time features for slowloris attacks and distributes attention across CAN ID and data fields in vehicle data.
Load-bearing premise
The load-bearing premise is that the random train/test split of CICIDS2017 does not put near-duplicate flows from the same attack session into both training and test; the paper never describes how the split was made, and CICIDS2017 captures several days of traffic in which flows from one attack are highly similar, so if that assumption fails the perfect macro-F1 numbers would collapse.
Editorial extensions
If this is right
- If NIDS-GPT's perfect macro-F1 holds at a 0.0002 imbalance ratio, intrusion detection can be built without resampling, GAN augmentation, or cost-sensitive reweighting, since the model learns rare classes directly from raw numeric tokens.
- A one-shot macro-F1 of 0.84 with one training sample per class would let defenders seed a detector for a newly observed attack type from a single labeled flow.
- Since the same architecture and hyperparameters transfer unchanged to CAN bus data with 1.00 macro-F1, the approach offers a single end-to-end recipe across network and vehicle-bus anomaly detection.
- Attention visualization that highlights inter-arrival time for slowloris and CAN ID for vehicle attacks could be used to generate human-readable explanations of why a packet was flagged.
- Larger model variants reach lower training negative log-likelihood, so if that trend continues, scaling the same tokenization and objective should further improve detection on harder, noisier traffic.
Reading between the lines
- A temporal or session-grouped split is the natural stress test for the reported perfect F1, and applying it would show whether the result transfers to deployment where future traffic is unseen.
- The digit-as-word representation plus numeric-position embeddings could generalize beyond packets to other structured numeric records, such as sensor logs, telemetry, or protocol payloads, where the relative place of a digit carries meaning.
- Because the label is just the last token, the objective can scale to new label vocabularies without changing the loss; appending textual or hierarchical labels could turn the same model into an open-set or zero-shot detector.
- One could test the method on attack types absent from training to see whether next-token surprise, not only the label readout, flags genuinely novel traffic; that would turn the language model into a true anomaly detector rather than a classifier.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes NIDS-GPT, a GPT-style causal language model for network intrusion detection. The core idea is to tokenize each digit of each numeric packet field as an independent token, reverse each field so that least-significant digits appear at consistent positions, and insert dynamic separator tokens that encode field boundaries and field order. The model uses a sum of word, numeric-position, and field-position embeddings and is trained with a next-token objective in which the packet label is appended as the final token. On CICIDS2017 the authors report 1.00 precision, recall, and F1 at imbalance ratios of 0.001, 0.0005, and 0.0002, 0.84 macro-F1 in a one-shot setting, and 1.00 on the car-hacking CAN dataset; attention visualizations are presented as interpretability evidence.
Significance. If the empirical claims were substantiated, the paper would be a notable empirical contribution: a digit-level representation for network flows, a custom tokenizer with field reversal and dynamic separators, and a next-token training objective that integrates classification into language modeling are all reasonable design ideas, and the one-shot result would be striking. The authors also provide model configuration tables and a code repository, which is helpful for reproducibility. However, the central 100% F1 result is currently unverifiable because the evaluation omits the train/test splitting procedure, uses a support-weighted average mislabeled as macro-F1, and drops entire attack classes in some splits. These issues are load-bearing for the headline claim, so the paper needs substantial revision before the results can be accepted.
major comments (4)
- [Section 4.1, Table 3] The manuscript never describes how the CICIDS2017 train and test sets were constructed; Table 3 reports only sample counts. CICIDS2017 is a five-day capture in which flows from the same attack session are highly correlated, and if the split is a random shuffle of individual flows, near-identical records can appear on both sides. With only 13–29 attack training samples per class and 4–20 test samples, perfect test recall in Tables 4–6 and the one-shot result in Table 7 would then reflect memorization rather than generalization. The authors must specify the exact split procedure, use a session-disjoint or temporally ordered split, and report per-class confusion matrices and variance over repeated splits.
- [Section 3.5, Eq. (14)] Eq. (14) defines 'Macro-F1' as sum_k (n_k/N) F1_k, which is a support-weighted average, not a macro average. Under the extreme class imbalance of Section 4.1, this quantity is dominated by the BENIGN class, so a value rounding to 1.00 at two decimals does not demonstrate perfect detection of the rare attack classes. Since Tables 4–6 label the reported rows as 'macro avg', the authors should either compute the unweighted macro-F1 and report per-class precision/recall for each minority class, or rename the metric and state clearly that it is support-weighted.
- [Section 4.1, Table 3 and Tables 4–6] The class set is not held constant across the three imbalance ratios: the Sql Injection class has zero training and test samples at ratios 0.001 and 0.0005, and the Infiltration class has zero samples at 0.001. Thus the '1.00' results in Tables 4 and 5 are computed over a different set of classes than the result in Table 6, and the macro-averaged scores are not directly comparable across ratios. The authors should either include all classes in every split or explicitly state which classes are excluded and justify why.
- [Section 4.1, Tables 4–7] No details are given for how the baseline methods (ET, RandomForest, optimized_RF, CNN_BiLSTM) were trained and tuned, whether they used the same train/test splits, or how many random repeats were performed; only the one-shot experiment is described as an average of 10 runs, and no standard deviation is reported. Without this information, the claim that NIDS-GPT 'far surpasses' the baselines cannot be verified, and the possibility that the baselines were undertuned cannot be excluded. Please report the exact protocol, baseline hyperparameters, and repeated-run statistics.
minor comments (5)
- [Abstract] The abstract reports '100% accuracy' while the experiments report precision, recall, and F1-score; please use consistent metric terminology throughout.
- [Section 3.2] The sentence 'the most significant digit (ones place) always appears in a fixed position' is internally contradictory: reversing the string puts the ones digit first, which is the least significant digit. Please correct the wording.
- [Section 3.4, Eq. (10)] The notation p_l is used in Eq. (10) but is not defined before the equation; please define it explicitly as the model's predicted distribution over the label token at the label position.
- [Section 4.1 and Figure 4] The training log in Figure 4 lacks axis labels and a legend, making it difficult to tell which curve corresponds to which model size and what quantity is plotted; please add these details.
- [Section 5] The conclusion refers to the 'OTIDS' dataset, but the experiments in Section 4.2 use the car-hacking dataset; please align the terminology.
Circularity Check
No circular derivation: reported results are empirical held-out comparisons; split and metric caveats are validity concerns, not circularity.
full rationale
The paper contains no derivation chain in which an output quantity is defined from the quantity it claims to predict. The method is a GPT-style next-token model (Eq. 9) with the packet label appended as the final token; training and evaluation are separate, with reported Precision/Recall/F1 values taken from held-out test rows (Tables 3-7). No fitted parameter is renamed as a prediction, no benchmark number is reconstructed from the model's own training objective, and the cited related work is external rather than load-bearing self-citation. The attention-weight discussion (Section 4.3) is post hoc and does not feed back into the reported metrics. The two serious weaknesses in this paper are validity concerns rather than circularity: the CICIDS2017 train/test split is never described (Section 4.1), so temporally correlated flows may be shared between train and test; and Eq. (14) calls a class-frequency-weighted average "Macro-F1," which can inflate the displayed value under extreme imbalance. Neither issue makes the reported numbers equivalent by construction to the inputs of a derivation, so the circularity score is 0.
Assumptions & free parameters
free parameters (6)
- sequence length L =
256
- embedding dimension d =
128 (base), 256 (small), 512 (middle)
- number of transformer layers =
6/8/10
- learning rate =
1e-4 (3e-5 for middle)
- MLP ratio =
8/3
- initial separator token S0 =
not stated
assumptions (6)
- domain assumption CICIDS2017 flow features can be treated as a sequence of numeric fields without significant information loss.
- domain assumption Random split of flows into train/test does not create temporal leakage.
- domain assumption Baseline methods (ET, RF, optimized_RF, CNN_BiLSTM) are reasonably tuned.
- domain assumption One training sample per attack class is representative of the class in the one-shot experiment.
- ad hoc to paper Reversing each numeric field preserves semantics needed for detection.
- ad hoc to paper Dynamic separator tokens encode field order and improve structure learning.
Cite this review
Pith. "Pith review of Take Package as Language: Anomaly Detection Using Transformer." pith.science (2026). https://pith.science/paper/IJGOUHND
@misc{pith2026241204473,
author = {Pith},
title = {Pith review of: Take Package as Language: Anomaly Detection Using Transformer},
year = {2026},
howpublished = {\url{https://pith.science/paper/IJGOUHND}},
note = {Machine review of arXiv:2412.04473}
}
read the original abstract
Network data packet anomaly detection faces numerous challenges, including exploring new anomaly supervision signals, researching weakly supervised anomaly detection, and improving model interpretability. This paper proposes NIDS-GPT, a GPT-based causal language model for network intrusion detection. Unlike previous work, NIDS-GPT innovatively treats each number in the packet as an independent "word" rather than packet fields, enabling a more fine-grained data representation. We adopt an improved GPT-2 model and design special tokenizers and embedding layers to better capture the structure and semantics of network data. NIDS-GPT has good scalability, supports unsupervised pre-training, and enhances model interpretability through attention weight visualization. Experiments on the CICIDS2017 and car-hacking datasets show that NIDS-GPT achieves 100\% accuracy under extreme imbalance conditions, far surpassing traditional methods; it also achieves over 90\% accuracy in one-shot learning. These results demonstrate NIDS-GPT's excellent performance and potential in handling complex network anomaly detection tasks, especially in data-imbalanced and resource-constrained scenarios. The code is available at \url{https://github.com/woshixiaobai2019/nids-gpt.gi
Figures
Figures from the paper (4 more)
Reference graph
Works this paper leans on
-
[1]
A systematic literature review for network intrusion detection system (ids)
Oluwadamilare Harazeem Abdulganiyu, Taha Ait Tchakoucht, and Yakub Kayode Saheed. A systematic literature review for network intrusion detection system (ids). International journal of information security , 22(5):1125– 1162, 2023
work page 2023
-
[2]
Jyoti Verma, Abhinav Bhandari, and Gurpreet Singh. inids: Swot analysis and tows inferences of state-of-the-art nids solutions for the development of intelligent network intrusion detection system. Computer Communications, 195:227–247, 2022
work page 2022
-
[3]
Toward generating a new intrusion detection dataset and intrusion traffic characterization
Iman Sharafaldin, Arash Habibi Lashkari, Ali A Ghorbani, et al. Toward generating a new intrusion detection dataset and intrusion traffic characterization. ICISSp, 1:108–116, 2018
2018
-
[4]
In-vehicle network intrusion detection using deep convolu- tional neural network
Hyun Min Song, Jiyoung Woo, and Huy Kang Kim. In-vehicle network intrusion detection using deep convolu- tional neural network. V ehicular Communications, 21:100198, 2020
work page 2020
-
[5]
Deep learning for anomaly detection: A review
Guansong Pang, Chunhua Shen, Longbing Cao, and Anton Van Den Hengel. Deep learning for anomaly detection: A review. ACM computing surveys (CSUR), 54(2):1–38, 2021
2021
-
[6]
Neha Gupta, Vinita Jindal, and Punam Bedi. Cse-ids: Using cost-sensitive deep learning and ensemble algorithms to handle class imbalance in network-based intrusion detection systems. Computers & Security , 112:102499, 2022
work page 2022
-
[7]
Intrusion detection system using pca with random forest approach
Subhash Waskle, Lokesh Parashar, and Upendra Singh. Intrusion detection system using pca with random forest approach. In 2020 International Conference on Electronics and Sustainable Communication Systems (ICESC) , pages 803–808. IEEE, 2020
work page 2020
-
[8]
Optimization of ids using filter-based feature selection and machine learning algorithms
Neha Sharma, Harsh Vardhan Bhandari, Narendra Singh Yadav, and HVJ Shroff. Optimization of ids using filter-based feature selection and machine learning algorithms. Int. J. Innov. Technol. Explor . Eng, 10(2):96–102, 2020. 13 A PREPRINT - DECEMBER 9, 2024
work page 2020
Show all 30 references
-
[9]
Intrusion detection system after data augmentation schemes based on the vae and cvae
Chang Liu, Ruslan Antypenko, Iryna Sushko, and Oksana Zakharchenko. Intrusion detection system after data augmentation schemes based on the vae and cvae. IEEE Transactions on Reliability, 71(2):1000–1010, 2022
2022
-
[10]
Gan-based imbalanced data intrusion detection system
JooHwa Lee and KeeHyun Park. Gan-based imbalanced data intrusion detection system. Personal and Ubiquitous Computing, 25(1):121–128, 2021
2021
-
[11]
Intrusion detection system based on one-class support vector machine and gaussian mixture model
Chao Wang, Yunxiao Sun, Sicai Lv, Chonghua Wang, Hongri Liu, and Bailing Wang. Intrusion detection system based on one-class support vector machine and gaussian mixture model. Electronics, 12(4):930, 2023
2023
-
[12]
Improving language understanding by generative pre-training
Alec Radford. Improving language understanding by generative pre-training. 2018
2018
-
[13]
Toutanova
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina N. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. 2018
2018
-
[14]
Language models are few-shot learners
Tom B Brown. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020
2005 arXiv
-
[15]
Emergent abilities of large language models
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682, 2022
2022 arXiv
-
[16]
A transfer learning and optimized cnn based intrusion detection system for internet of vehicles
Li Yang and Abdallah Shami. A transfer learning and optimized cnn based intrusion detection system for internet of vehicles. In ICC 2022-IEEE International Conference on Communications , pages 2774–2779. IEEE, 2022
2022
-
[17]
Cnn-based network intrusion detection against denial-of-service attacks
Jiyeon Kim, Jiwon Kim, Hyunjung Kim, Minsun Shim, and Eunjung Choi. Cnn-based network intrusion detection against denial-of-service attacks. Electronics, 9(6):916, 2020
2020
-
[18]
A novel two-stage deep learning model for network intrusion detection: Lstm-ae
Vanlalruata Hnamte, Hong Nhung-Nguyen, Jamal Hussain, and Yong Hwa-Kim. A novel two-stage deep learning model for network intrusion detection: Lstm-ae. Ieee Access, 11:37131–37148, 2023
2023
-
[19]
An effective recurrent neural network (rnn) based intrusion detection via bi-directional long short-term memory
S Sivamohan, SS Sridhar, and S Krishnaveni. An effective recurrent neural network (rnn) based intrusion detection via bi-directional long short-term memory. In 2021 international conference on intelligent technologies (CONIT) , pages 1–5. IEEE, 2021
2021
-
[20]
Canbert: A language-based intrusion detection model for in-vehicle networks
Ebelechukwu Nwafor and Habeeb Olufowobi. Canbert: A language-based intrusion detection model for in-vehicle networks. In 2022 21st IEEE International Conference on Machine Learning and Applications (ICMLA) , pages 294–299. IEEE, 2022
2022
-
[21]
Neural machine translation of rare words with subword units
Rico Sennrich. Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07909, 2015
2015 arXiv
-
[22]
Roformer: Enhanced transformer with rotary position embedding
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024
2024
-
[23]
Can-bert do it? controller area network intrusion detection system based on bert language model
Natasha Alkhatib, Maria Mushtaq, Hadi Ghauch, and Jean-Luc Danger. Can-bert do it? controller area network intrusion detection system based on bert language model. In 2022 IEEE/ACS 19th International Conference on Computer Systems and Applications (AICCSA) , pages 1–8. IEEE, 2022
2022
-
[24]
Intrusion detection method using bi-directional gpt for in-vehicle controller area networks
Minki Nam, Seungyoung Park, and Duk Soo Kim. Intrusion detection method using bi-directional gpt for in-vehicle controller area networks. IEEE Access, 9:124931–124944, 2021
2021
-
[25]
Language models can improve event prediction by few-shot abductive reasoning
Xiaoming Shi, Siqiao Xue, Kangrui Wang, Fan Zhou, James Zhang, Jun Zhou, Chenhao Tan, and Hongyuan Mei. Language models can improve event prediction by few-shot abductive reasoning. Advances in Neural Information Processing Systems, 36, 2024
2024
-
[26]
Time-llm: Time series forecasting by reprogramming large language models
Ming Jin, Shiyu Wang, Lintao Ma, Zhixuan Chu, James Y Zhang, Xiaoming Shi, Pin-Yu Chen, Yuxuan Liang, Yuan-Fang Li, Shirui Pan, et al. Time-llm: Time series forecasting by reprogramming large language models. arXiv preprint arXiv:2310.01728, 2023
-
[27]
Glu variants improve transformer
Noam Shazeer. Glu variants improve transformer. arXiv preprint arXiv:2002.05202, 2020
2002 arXiv
-
[28]
Self-supervised Pre-training on LSTM and Transformer Models for Network Intrusion Detection
Jonas Ferdigg. Self-supervised Pre-training on LSTM and Transformer Models for Network Intrusion Detection . PhD thesis, Technische Universität Wien, 2022
2022
-
[29]
Synthe- sis of a machine learning model for detecting computer attacks based on the cicids2017 dataset
Maxim Nikolaevich Goryunov, Andrey Georgievich Matskevich, and Dmitry Aleksandrovich Rybolovlev. Synthe- sis of a machine learning model for detecting computer attacks based on the cicids2017 dataset. Proceedings of the Institute for System Programming of the RAS , 32(5):81–94, 2020
2020
-
[30]
Deep learning applications for intrusion detection in network traffic
Alexander Igorevich Get’man, Maxim Nikolaevich Goryunov, Andrey Georgievich Matskevich, Dmitry Aleksan- drovich Rybolovlev, and Anastasiya Grigorevna Nikolskaya. Deep learning applications for intrusion detection in network traffic. Proceedings of the Institute for System Prog...
2023
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.