{"id":"eaa14d9d-3cdf-4115-9f7c-aabd9552317a","arxiv_id":"2412.04473","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"A digit-level causal language model for network flows reports 100% macro-F1 under extreme class imbalance and 0.84 one-shot F1 on CICIDS2017.","lead":"This paper proposes NIDS-GPT, a GPT-style model that reads each digit in a network flow as a separate 'word' and predicts the next digit, with the flow's label appended as the final word. The authors report perfect macro-F1 on heavily imbalanced intrusion detection benchmarks and above 90% precision in one-shot settings, but the supporting experiments lack error bars and split details.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 100% macro-F1 result rests on an undescribed split of temporally correlated CICIDS2017 flows; with only 13–29 attack training samples, perfect test recall indicates likely train/test leakage.","rationale":"The reader's weakest-assumption analysis identifies the same load-bearing concern: the evaluation's validity hinges on an undescribed train/test split, and CICIDS2017's temporal structure makes leakage plausible. This is the most important threat to the central claim because the 100% scores are the paper's headline evidence; if the split leaks, the claim collapses regardless of the method's theoretical merit. The manuscript explicitly omits the split procedure, which is a missing support that should be flagged. The method itself — digit-level tokenization with a GPT-style next-token objective — is plausible and worth investigating, so the appropriate disposition remains CONDITIONAL pending a reproducible, leakage-free evaluation. The secondary metric concern (Eq. 14 calling a weighted average 'Macro-F1') further weakens the reported numbers but does not change the verdict; it reinforces the need for the authors to release code and exact evaluation details. No adjustment to the reader's verdict is needed.","tokens_in":10789,"tokens_out":7474,"duration_ms":79809,"concrete_test":"Reproduce the 0.0002 imbalance experiment (Table 6) using a temporal split of CICIDS2017: sort all flows by timestamp, take the first 60% for training and the last 40% for testing (or, better, group by 5-tuple so no connection straddles the boundary), keeping Table 3's class counts; compute unweighted macro-F1. If it is below 1.00, the original perfect scores depended on train/test correlation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim — 100% Precision/Recall/F1 at imbalance ratios 0.001–0.0002 — depends entirely on how the CICIDS2017 train/test split was performed, and the manuscript never describes it (Section 4.1, Table 3). With only 13–29 attack samples per class in training and 4–20 in testing, a clean split would require the model to generalize to unseen attack flows from extremely few examples. CICIDS2017 is a multi-day capture; flows from the same attack session share nearly all feature values except timestamp. If the split is a random shuffle of individual flows, the same sessions appear on both sides, so the model can memorize exact vectors and achieve perfect test recall without learning any general detector. The reported 1.00 in Tables 4–6 is therefore not evidence of the method's merit unless the split is shown to be temporally or session-disjoint. The one-shot result (Table 7) is subject to the same defect. A secondary concern: Eq. (14) defines 'Macro-F1' as a frequency-weighted average, so a single failed rare class yields ~0.999 and rounds to 1.00 at two decimals; true macro-F1 could be much lower. These issues make the headline result unverifiable from the paper alone.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes NIDS-GPT, a GPT-style causal language model for network intrusion detection. The core idea is to tokenize each digit of each numeric packet field as an independent token, reverse each field so that least-significant digits appear at consistent positions, and insert dynamic separator tokens that encode field boundaries and field order. The model uses a sum of word, numeric-position, and field-position embeddings and is trained with a next-token objective in which the packet label is appended as the final token. On CICIDS2017 the authors report 1.00 precision, recall, and F1 at imbalance ratios of 0.001, 0.0005, and 0.0002, 0.84 macro-F1 in a one-shot setting, and 1.00 on the car-hacking CAN dataset; attention visualizations are presented as interpretability evidence.","tokens_in":11095,"tokens_out":4421,"duration_ms":46419,"significance":"If the empirical claims were substantiated, the paper would be a notable empirical contribution: a digit-level representation for network flows, a custom tokenizer with field reversal and dynamic separators, and a next-token training objective that integrates classification into language modeling are all reasonable design ideas, and the one-shot result would be striking. The authors also provide model configuration tables and a code repository, which is helpful for reproducibility. However, the central 100% F1 result is currently unverifiable because the evaluation omits the train/test splitting procedure, uses a support-weighted average mislabeled as macro-F1, and drops entire attack classes in some splits. These issues are load-bearing for the headline claim, so the paper needs substantial revision before the results can be accepted.","major_comments":[{"comment":"The manuscript never describes how the CICIDS2017 train and test sets were constructed; Table 3 reports only sample counts. CICIDS2017 is a five-day capture in which flows from the same attack session are highly correlated, and if the split is a random shuffle of individual flows, near-identical records can appear on both sides. With only 13–29 attack training samples per class and 4–20 test samples, perfect test recall in Tables 4–6 and the one-shot result in Table 7 would then reflect memorization rather than generalization. The authors must specify the exact split procedure, use a session-disjoint or temporally ordered split, and report per-class confusion matrices and variance over repeated splits.","section":"Section 4.1, Table 3"},{"comment":"Eq. (14) defines 'Macro-F1' as sum_k (n_k/N) F1_k, which is a support-weighted average, not a macro average. Under the extreme class imbalance of Section 4.1, this quantity is dominated by the BENIGN class, so a value rounding to 1.00 at two decimals does not demonstrate perfect detection of the rare attack classes. Since Tables 4–6 label the reported rows as 'macro avg', the authors should either compute the unweighted macro-F1 and report per-class precision/recall for each minority class, or rename the metric and state clearly that it is support-weighted.","section":"Section 3.5, Eq. (14)"},{"comment":"The class set is not held constant across the three imbalance ratios: the Sql Injection class has zero training and test samples at ratios 0.001 and 0.0005, and the Infiltration class has zero samples at 0.001. Thus the '1.00' results in Tables 4 and 5 are computed over a different set of classes than the result in Table 6, and the macro-averaged scores are not directly comparable across ratios. The authors should either include all classes in every split or explicitly state which classes are excluded and justify why.","section":"Section 4.1, Table 3 and Tables 4–6"},{"comment":"No details are given for how the baseline methods (ET, RandomForest, optimized_RF, CNN_BiLSTM) were trained and tuned, whether they used the same train/test splits, or how many random repeats were performed; only the one-shot experiment is described as an average of 10 runs, and no standard deviation is reported. Without this information, the claim that NIDS-GPT 'far surpasses' the baselines cannot be verified, and the possibility that the baselines were undertuned cannot be excluded. Please report the exact protocol, baseline hyperparameters, and repeated-run statistics.","section":"Section 4.1, Tables 4–7"}],"minor_comments":[{"comment":"The abstract reports '100% accuracy' while the experiments report precision, recall, and F1-score; please use consistent metric terminology throughout.","section":"Abstract"},{"comment":"The sentence 'the most significant digit (ones place) always appears in a fixed position' is internally contradictory: reversing the string puts the ones digit first, which is the least significant digit. Please correct the wording.","section":"Section 3.2"},{"comment":"The notation p_l is used in Eq. (10) but is not defined before the equation; please define it explicitly as the model's predicted distribution over the label token at the label position.","section":"Section 3.4, Eq. (10)"},{"comment":"The training log in Figure 4 lacks axis labels and a legend, making it difficult to tell which curve corresponds to which model size and what quantity is plotted; please add these details.","section":"Section 4.1 and Figure 4"},{"comment":"The conclusion refers to the 'OTIDS' dataset, but the experiments in Section 4.2 use the car-hacking dataset; please align the terminology.","section":"Section 5"}],"recommendation":"major_revision","confidential_remarks":"The paper's central empirical claim is not currently reproducible from the manuscript. The most important missing piece is the train/test split procedure for CICIDS2017, and the support-weighted metric mislabeled as macro-F1 further weakens the reported 1.00 values. I recommend a major revision focused on these evaluation issues rather than on the architectural proposal itself, which is reasonable and potentially interesting. Please also check that the class composition is consistent across imbalance-ratio comparisons."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Honest take: the tokenization design is the real contribution here. Digit-level tokenization with field reversal and dynamic separators is a genuine new way to represent structured numeric data for a causal LM, and the embedding strategy (digit position plus field position) is thoughtful. The method section is clear enough that someone could reimplement the tokenizer. The attention visualizations are a nice bonus, even if qualitative.\n\nThe problem is the evaluation. The paper never says how the CICIDS2017 split was constructed. Table 3 just gives counts. CICIDS2017 is a multi-day capture; flows from the same attack session are near-duplicates. With only 13–29 attack training samples per class, a random shuffle of flows would put the same sessions in both train and test, and perfect test recall becomes a test of memory, not generalization. The stress-test's main concern lands. You need session-disjoint or timestamp-based splits, and that must be stated explicitly.\n\nThere's a second, independent problem: Eq. (14) labels a frequency-weighted average as 'Macro-F1'. True macro-F1 is an unweighted average. With 99% benign test samples, a single failed rare class changes the weighted F1 by about 0.0002, which rounds to 1.00 at two decimals. So Tables 4–6 could show 1.00 while a class completely fails. The evaluation needs per-class results or a proper unweighted macro-F1. Add error bars; the one-shot result is averaged over 10 runs but no standard deviation is reported.\n\nThe baselines are also taken from other papers, not re-tuned. That's not fatal, but it makes the comparison weaker.\n\nTaken together, I cannot trust the headline 100% number. The method might still be good — the car-hacking results are less prone to temporal leakage, and the tokenizer has merit. The code is available, so a referee can check. I'd send this to peer review, but only with a major-revision bar: require a clear split description, per-class metrics, true macro-F1, error bars, and baseline re-runs. The underlying idea is worth the effort.","headline":"The digit-level tokenizer is a real idea; the evaluation's unstated split and mislabeled metric make the headline 100% F1 unsupported.","tokens_in":11566,"tokens_out":3363,"would_cite":false,"duration_ms":33559,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Packet-as-language GPT reports 1.00 macro-F1 on rare attacks.","keywords":["network intrusion detection","anomaly detection","causal language model","packet tokenization","imbalanced learning","one-shot learning","transfer learning","attention interpretability"],"falsifier":"A concrete check: re-run the CICIDS2017 experiment with a temporal split that trains on flows captured on the first days and tests on flows captured on later days, or group flows by attack session or timestamp so no session spans the split, and report macro-F1 at the same three imbalance ratios. If macro-F1 drops substantially below 1.00 under that split, the perfect scores are an artifact of train/test similarity rather than a property of the packet-as-language objective.","tokens_in":10604,"feed_emoji":"🛡️","tokens_out":9564,"duration_ms":88509,"temperature":0.7,"pith_summary":"Network intrusion detection usually treats packet fields as coarse features and struggles when attack samples are vanishingly rare. This paper tries to establish a different route: treat every digit inside a packet as a token of a language, reverse each numeric field so unit digits always occupy the same position, mark field boundaries with dynamic separator tokens, and train a GPT-style causal decoder to predict the next token, with the packet label appended as the final token. On the CICIDS2017 benchmark at imbalance ratios of 0.001, 0.0005, and 0.0002 the method reports macro average Precision, Recall, and F1 of 1.00, and 0.91/0.82/0.84 in one-shot learning with a single training sample per class; it also reports 1.00 on a CAN car-hacking dataset without changing the model. If these results hold, a single language-modelling recipe could handle extreme class imbalance, few-shot adaptation, and cross-domain transfer for packet anomaly detection.","feed_headline":"Packet-as-language GPT scores 1.00 macro-F1 on rare attacks","feed_subtitle":"Reading each digit as a token, the same model also handles one-shot and CAN-bus attacks.","key_machinery":"The load-bearing machinery is a deliberately unusual tokenizer plus a three-part embedding. The tokenizer reverses the character order of every numeric field, converts each digit into its own token, inserts separators $S_i = S_0 + i - 1$ that both mark field boundaries and encode field order, and appends the class label as the final integer token. Each token then receives the sum of three embeddings: a word embedding for its digit value, a numeric-position embedding indicating whether the digit is in the ones, tens, or hundreds place of its field, and a field-position embedding indicating which packet field it came from. A GPT-2-style causal Transformer decoder is trained with next-token negative log-likelihood over all tokens, and at inference the model reads out the label by taking the argmax of the probability at the label position. This design is what lets the model treat classification as language modelling and, the paper argues, is why it survives extreme imbalance.","core_discovery":"On the paper's own terms, the central discovery is that a causal language model trained to predict the next token of a packet, where each packet is serialized digit-by-digit with reversed numeric fields and the class label placed as the last token, learns to classify network traffic so well that it attains 1.00 macro Precision, Recall, and F1 on CICIDS2017 at imbalance ratios of 0.001, 0.0005, and 0.0002, outperforming Extra Trees, Random Forest, an optimized random forest, and a CNN-BiLSTM baseline. In a one-shot protocol averaged over ten random draws, it reports 0.91 precision, 0.82 recall, and 0.84 macro-F1 with one training sample per class. The same model, retrained with the same hyperparameters on the car-hacking CAN bus dataset, reports 1.00 on all three metrics, matching a transfer-CNN baseline. Attention-weight visualizations show the model attends to inter-arrival time features for slowloris attacks and distributes attention across CAN ID and data fields in vehicle data.","pith_inferences":["A temporal or session-grouped split is the natural stress test for the reported perfect F1, and applying it would show whether the result transfers to deployment where future traffic is unseen.","The digit-as-word representation plus numeric-position embeddings could generalize beyond packets to other structured numeric records, such as sensor logs, telemetry, or protocol payloads, where the relative place of a digit carries meaning.","Because the label is just the last token, the objective can scale to new label vocabularies without changing the loss; appending textual or hierarchical labels could turn the same model into an open-set or zero-shot detector.","One could test the method on attack types absent from training to see whether next-token surprise, not only the label readout, flags genuinely novel traffic; that would turn the language model into a true anomaly detector rather than a classifier."],"forward_implications":["If NIDS-GPT's perfect macro-F1 holds at a 0.0002 imbalance ratio, intrusion detection can be built without resampling, GAN augmentation, or cost-sensitive reweighting, since the model learns rare classes directly from raw numeric tokens.","A one-shot macro-F1 of 0.84 with one training sample per class would let defenders seed a detector for a newly observed attack type from a single labeled flow.","Since the same architecture and hyperparameters transfer unchanged to CAN bus data with 1.00 macro-F1, the approach offers a single end-to-end recipe across network and vehicle-bus anomaly detection.","Attention visualization that highlights inter-arrival time for slowloris and CAN ID for vehicle attacks could be used to generate human-readable explanations of why a packet was flagged.","Larger model variants reach lower training negative log-likelihood, so if that trend continues, scaling the same tokenization and objective should further improve detection on harder, noisier traffic."],"supporting_citations":[{"why":"It supplies the CICIDS2017 dataset and its attack taxonomy used in the main imbalance experiments.","marker":"[3]"},{"why":"It supplies the car-hacking CAN dataset used for the transfer experiment.","marker":"[4]"},{"why":"It provides the generative pre-training objective that NIDS-GPT adapts to packet tokens.","marker":"[12]"},{"why":"It motivates the GPT-2-style decoder architecture and scaling behavior adopted by the model.","marker":"[14]"},{"why":"It supplies the transfer-CNN baselines compared on the car-hacking dataset.","marker":"[16]"},{"why":"It defines the field-level BERT packet modeling approach that NIDS-GPT's digit-level tokenization is contrasted with.","marker":"[20]"},{"why":"It provides the imbalance-ratio evaluation protocol that Section 4.1 extends to ratios of 0.001, 0.0005, and 0.0002.","marker":"[28]"},{"why":"It supplies the optimized_RF method compared in the imbalance and one-shot tables.","marker":"[29]"},{"why":"It supplies the CNN_BiLSTM method compared in the imbalance and one-shot tables.","marker":"[30]"}],"fun_headline_variants":["Packet digits as words: GPT detects rare attacks at perfect F1","Tokenizing packets digit-by-digit yields 100% F1 under imbalance","Transformer reads packet numbers as language, scores perfect F1","GPT-2 packet anomaly detector hits 1.00 F1 on rare classes"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the random train/test split of CICIDS2017 does not put near-duplicate flows from the same attack session into both training and test; the paper never describes how the split was made, and CICIDS2017 captures several days of traffic in which flows from one attack are highly similar, so if that assumption fails the perfect macro-F1 numbers would collapse.","fun_headline_variants_meta":{"raw":{"variants":["Packet digits as words: GPT detects rare attacks at perfect F1","Tokenizing packets digit-by-digit yields 100% F1 under imbalance","Transformer reads packet numbers as language, scores perfect F1","GPT-2 packet anomaly detector hits 1.00 F1 on rare classes"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000234,"raw_usage":{"total_tokens":1513,"prompt_tokens":980,"completion_tokens":533,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":596,"completion_tokens_details":{"reasoning_tokens":456}},"tokens_in":596,"tokens_out":533,"duration_ms":6067,"temperature":1.0,"reasoning_tokens":456,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T20:11:45.822225+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete check: re-run the CICIDS2017 experiment with a temporal split that trains on flows captured on the first days and tests on flows captured on later days, or group flows by attack session or timestamp so no session spans the split, and report macro-F1 at the same three imbalance ratios. If macro-F1 drops substantially below 1.00 under that split, the perfect scores are an artifact of train/test similarity rather than a property of the packet-as-language objective.","supporting_citations":[{"cited_title":"In-vehicle network intrusion detection using deep convolu- tional neural network","cited_arxiv_id":null,"evidence_quote":"It supplies the car-hacking CAN dataset used for the transfer experiment."},{"cited_title":"A transfer learning and optimized cnn based intrusion detection system for internet of vehicles","cited_arxiv_id":null,"evidence_quote":"It supplies the transfer-CNN baselines compared on the car-hacking dataset."},{"cited_title":"Canbert: A language-based intrusion detection model for in-vehicle networks","cited_arxiv_id":null,"evidence_quote":"It defines the field-level BERT packet modeling approach that NIDS-GPT's digit-level tokenization is contrasted with."},{"cited_title":"Self-supervised Pre-training on LSTM and Transformer Models for Network Intrusion Detection","cited_arxiv_id":null,"evidence_quote":"It provides the imbalance-ratio evaluation protocol that Section 4.1 extends to ratios of 0.001, 0.0005, and 0.0002."},{"cited_title":"Synthe- sis of a machine learning model for detecting computer attacks based on the cicids2017 dataset","cited_arxiv_id":null,"evidence_quote":"It supplies the optimized_RF method compared in the imbalance and one-shot tables."},{"cited_title":"Deep learning applications for intrusion detection in network traffic","cited_arxiv_id":null,"evidence_quote":"It supplies the CNN_BiLSTM method compared in the imbalance and one-shot tables."}],"review_version":1}