Pith. sign in

REVIEW 3 major objections 12 references

A Protocol-Language Model for Network Intrusion (Without Deep Packet Inspection)

T0 review · 3 major / 0 minor · reviewed 2026-06-28 · grok-4.3

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A language model trained solely on benign network flows detects attacks through elevated perplexity on their metadata sequences.

desk verdict RWKV on Monday benign flows gives a clean perplexity separation from attacks but the day-specific training leaves the attack-specific claim unproven without checks on other benign days. read the letter →

arxiv 2606.00155 v1 pith:THS2M4BD submitted 2026-05-29 cs.CR cs.AI

Vivek Kumar Sharma This is my paper · ORCID
classification cs.CRcs.AI
keywords networkintrusiondetectionlanguagemodelingperplexityscoringflowmetadatastatespacemodelencryptionagnosticunsupervisedanomaly
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper shows that packet metadata sequences from normal traffic form a learnable grammar that a state-space model can capture from unlabeled examples alone. Attack flows then stand out because they produce higher perplexity under the same model, yielding strong separation without any attack examples seen during pretraining. The separation depends on the model's recurrent pretraining, since an LSTM baseline on identical data collapses to a trivial predictor. Because only header fields are used, the method applies directly to encrypted protocols.

What carries the argument

RWKV-4 state-space model trained as a causal language model on tokenized sequences of packet length, inter-arrival time, TTL, flags and hashed ports, with per-flow perplexity used as the detection score.

What would settle it

A collection of attack flows that receive perplexity scores comparable to the benign validation set under the trained model.

Watch

Extended reading notes

Core claim

Benign network flows possess a statistically consistent structure in L3/L4 metadata that a causal language model can learn, and attack flows reliably violate that structure enough for per-flow perplexity to serve as an effective unsupervised detector.

Load-bearing premise

Attack flows will produce reliably higher perplexity than benign flows rather than simply different but still predictable patterns.

Editorial extensions

If this is right

  • Detection runs at line rate because the model supports O(T) recurrent inference without buffering entire flows.
  • The detector remains effective on TLS 1.3 and QUIC traffic since it never examines payloads.
  • Adding a small amount of labeled data through supervised fine-tuning further improves precision and recall.
  • Architectures without the causal pretraining bias fail to produce the separation on the same metadata sequences.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Similar unsupervised pretraining on normal data could apply to anomaly detection in other sequential domains such as system logs or sensor streams.
  • The approach suggests that many signature-based intrusion systems might be replaceable by grammar-violation scores when representative normal traffic is available.
  • Real-time streaming becomes feasible in high-speed environments where flow reassembly is impractical.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 0 minor

Summary. The paper proposes PLM-NIDS, a RWKV-4 state-space model trained only on unlabelled benign network flows (344,232 Monday flows) using L3/L4 metadata sequences (packet length, inter-arrival time, TTL, flags, hashed ports) as tokens. It claims three results: (1) benign traffic has learnable structure, shown by causal LM validation loss of 0.204; (2) attack flows violate this grammar, yielding per-flow perplexity separation with PR-AUC 0.93 in a zero-label setting; (3) the separation is architecturally nontrivial, as an LSTM on the same sequences yields ROC-AUC ~0.50. Supervised fine-tuning raises PR-AUC to 0.94 and ROC-AUC to 0.75; the O(T) recurrent inference supports streaming detection without DPI or flow buffering.

Significance. If the central claims hold after addressing the representativeness issue, the work would be significant for encryption-agnostic NIDS: it demonstrates that next-token prediction on metadata alone can produce a useful anomaly score without attack labels, with a concrete LSTM contrast showing the value of the RWKV inductive bias. The streaming capability and zero-label framing are operationally relevant strengths.

major comments (3)
  1. [Abstract] Abstract, claim (2): The reported PR-AUC = 0.93 for attack separation by perplexity is load-bearing for the zero-label contribution, yet the manuscript provides no perplexity statistics, ROC/PR curves, or quantitative comparison on held-out benign flows from Tuesday–Friday. Without this, it is impossible to rule out that elevated perplexity reflects day-of-week distribution shift rather than attack-specific grammar violation.
  2. [Dataset and experimental setup] Dataset and experimental setup (assumed §3–4): The abstract states validation loss 0.204, PR-AUC 0.93, and LSTM baseline 0.50 but supplies no description of the full dataset source, train/test split procedure, tokenization vocabulary construction, sequence length handling, or statistical significance testing (e.g., confidence intervals on PR-AUC). These omissions make the numerical claims impossible to reproduce or assess for robustness.
  3. [Abstract] Abstract, fine-tuning paragraph: The claim that supervised fine-tuning raises PR-AUC from 0.93 to 0.94 is presented without an ablation isolating the contribution of the zero-shot perplexity score versus the added labels, leaving unclear whether the core unsupervised result already suffices or whether labels are required for operational performance.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for their careful reading and constructive comments on reproducibility and the need for additional controls. We address each point below and will revise the manuscript accordingly.

read point-by-point responses

  1. Referee: [Abstract] Abstract, claim (2): The reported PR-AUC = 0.93 for attack separation by perplexity is load-bearing for the zero-label contribution, yet the manuscript provides no perplexity statistics, ROC/PR curves, or quantitative comparison on held-out benign flows from Tuesday–Friday. Without this, it is impossible to rule out that elevated perplexity reflects day-of-week distribution shift rather than attack-specific grammar violation.

    Authors: We agree this analysis is necessary. The revised manuscript will add perplexity histograms, ROC/PR curves, and quantitative tables comparing held-out benign flows from Tuesday–Friday (which exhibit low perplexity matching the Monday validation set) against attack flows. This will demonstrate that the separation arises from grammar violation rather than temporal shift. revision: yes

  2. Referee: [Dataset and experimental setup] Dataset and experimental setup (assumed §3–4): The abstract states validation loss 0.204, PR-AUC 0.93, and LSTM baseline 0.50 but supplies no description of the full dataset source, train/test split procedure, tokenization vocabulary construction, sequence length handling, or statistical significance testing (e.g., confidence intervals on PR-AUC). These omissions make the numerical claims impossible to reproduce or assess for robustness.

    Authors: We acknowledge the current manuscript lacks these details. The revision will expand the experimental setup section to specify the CIC-IDS2017 source, the exact train/test split (Monday benign flows only for pre-training), vocabulary construction from L3/L4 metadata, sequence length handling, and bootstrap confidence intervals on all reported metrics. revision: yes

  3. Referee: [Abstract] Abstract, fine-tuning paragraph: The claim that supervised fine-tuning raises PR-AUC from 0.93 to 0.94 is presented without an ablation isolating the contribution of the zero-shot perplexity score versus the added labels, leaving unclear whether the core unsupervised result already suffices or whether labels are required for operational performance.

    Authors: We will add an explicit ablation comparing zero-shot perplexity scoring against the fine-tuned model. The revision will show that the unsupervised perplexity already reaches PR-AUC 0.93, with fine-tuning contributing only a marginal 0.01 gain, confirming the core zero-label result is sufficient for strong performance. revision: yes

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; derivation is self-contained

full rationale

The paper trains an RWKV-4 model solely on unlabeled benign Monday flows to minimize next-token prediction loss, then applies the resulting perplexity as an out-of-distribution score to attack flows. This is a standard unsupervised anomaly-detection construction that does not reduce to its inputs by definition: the training objective (benign LM loss) is independent of the evaluation metric (PR-AUC on attacks), and the LSTM baseline supplies an external architectural control. No self-citations, fitted parameters renamed as predictions, or uniqueness theorems appear in the derivation chain. The Monday-only training assumption is a potential generalization risk but is not a circularity.

Assumptions & free parameters 0 free parameters · 1 assumptions · 0 invented entities

Abstract-only; no explicit free parameters, axioms, or invented entities are stated beyond standard language-modeling assumptions.

assumptions (1)
  • domain assumption Benign network flows possess statistically consistent sequential structure that can be captured by next-token prediction loss
    Invoked in claim (1) to justify causal LM training on Monday flows

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Protocol-Language Model for Network Intrusion (Without Deep Packet Inspection)." pith.science (2026). https://pith.science/paper/THS2M4BD

@misc{pith2026260600155,
  author       = {Pith},
  title        = {Pith review of: A Protocol-Language Model for Network Intrusion (Without Deep Packet Inspection)},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/THS2M4BD}},
  note         = {Machine review of arXiv:2606.00155}
}
read the original abstract

Modern network intrusion detection systems (NIDS) are caught in a structural contradiction: the protocols carrying the highest threat intelligence are precisely those encrypted under TLS 1.3 and QUIC, where payload inspection yields nothing. We ask a simpler question -- what if the attack signature is not in the bytes, but in the rhythm? -- and answer it by treating network flows as a language whose grammar is written entirely in L3/L4 packet metadata: length, inter-arrival time, TTL, TCP flags, and hashed port numbers. We present PLM-NIDS, which proves three claims in sequence. (1) The grammar exists and is learnable: a RWKV-4 state-space model trained on 344,232 unlabelled Monday flows achieves a causal LM validation loss of 0.204, demonstrating that benign traffic has predictable, statistically consistent structure. (2) Attacks violate this grammar: the per-flow perplexity score cleanly separates benign from attack flows with PR-AUC = 0.93 using zero attack labels at training time. (3) This separation is architecturally nontrivial: an LSTM trained on identical token sequences degenerates to a majority-class predictor (ROC-AUC approximately 0.50, F1 = 0.91 by always predicting "attack"), proving that RWKV's causal pre-training provides an inductive bias unavailable to direct classifiers. Supervised fine-tuning further raises PR-AUC to 0.94 and ROC-AUC to 0.75, with a precision of 97.7% at the calibrated operating threshold. The RWKV backbone's O(T) recurrent inference enables per-packet streaming without flow buffering, making PLM-NIDS operationally viable at line rate. Because it reads only IP/TCP/UDP headers, it is inherently encryption-agnostic: TLS 1.3, QUIC, and future encrypted protocols are handled transparently.

Figures

Figures reproduced from arXiv: 2606.00155 by the authors.

Figure 1
Figure 1. Token vocabulary composition. All 227 tokens are derived exclusively from L3/L4 [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. Left: Token sequence length distributions per day (log-scale). Benign flows (Monday) concentrate at short lengths; DoS attacks (Wednesday) produce flows exceeding 1,000 tokens, making long-sequence modelling essential for full coverage. Right: Compute cost comparison. At the 95th-percentile flow length (521 tokens), RWKV requires 521 operations versus 271,441 for a Transformer—a 521× advantage that makes real-time s… view at source ↗
Figure 3
Figure 3. End-to-end PLM-NIDS inference pipeline. Each packet produces 9 header-only tokens processed one at a time by the RWKV recurrence. No payload byte enters the pipeline at any stage; the system can operate on a standard NIC tap with no DPI hardware. 4 Experiments Our experimental design is driven by the three claims in Section 1. To validate C1, we report Phase 1 training convergence. To validate C2, we report PLM-PPL … view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: Phase-1 (left): Causal LM convergence on benign Monday flows. Validation loss reaches 0.204 within 10 epochs, confirming that benign traffic has learnable, consistent gram￾matical structure. Phase-2 (right): Supervised fine-tuning converges at epoch 3 (early stop￾ping)…
Figure 5
Figure 5. Figure 5: Detection curves for PLM-CLS on the full 410,591-flow test set. The ROC-AUC of [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]
Figure 6
Figure 6. Figure 6: Left: Perplexity score distributions (PLM-PPL). Benign flows (blue) cluster at low perplexity; attack flows (red) shift right toward higher values. The vertical line marks the p95 calibrated threshold. The clear distributional separation provides visual confirmation of…
Figure 7
Figure 7. Figure 7: TPR and FPR versus decision threshold for PLM-PPL. The p95 operating point [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 8
Figure 8. Figure 8: Anomaly score distributions for all three modes. In PLM-PPL, the perplexity sepa [PITH_FULL_IMAGE:figures/full_fig_p018_8.png]
Figure 9
Figure 9. Figure 9: ROC curves across all scoring modes. PLM-CLS achieves the highest AUC (0.751), [PITH_FULL_IMAGE:figures/full_fig_p018_9.png]
Figure 10
Figure 10. Figure 10: Precision-Recall curves. The dashed line marks the 83% attack base rate (random [PITH_FULL_IMAGE:figures/full_fig_p018_10.png]
Figure 11
Figure 11. Figure 11: Normalised confusion matrices at the p95 calibrated threshold. All three modes [PITH_FULL_IMAGE:figures/full_fig_p019_11.png]
Figure 12
Figure 12. Figure 12: TPR and FPR as a function of the decision threshold for all three modes. Operators [PITH_FULL_IMAGE:figures/full_fig_p019_12.png]
Figure 13
Figure 13. Figure 13: Per-attack-category detection performance heatmap for PLM-PPL. DoS and Botnet [PITH_FULL_IMAGE:figures/full_fig_p020_13.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

12 extracted references · 2 canonical work pages

  1. [1]

    CICFlowMeter: Network traffic flow generator and analyser.https://www.unb.ca/cic/research/applications.html, 2017

    Canadian Institute for Cybersecurity. CICFlowMeter: Network traffic flow generator and analyser.https://www.unb.ca/cic/research/applications.html, 2017

    2017

  2. [2]

    HTTPS encryption on the web.https://transparencyreport.google

    Google LLC. HTTPS encryption on the web.https://transparencyreport.google. com/https/overview, 2024. Accessed: May 2026

    2024

  3. [3]

    Mamba: Linear-Time Sequence Modeling with Selective State Spaces

    Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023

    work page Pith review arXiv 2023

  4. [4]

    ET-BERT: A contextualized datagram representation with pre-training transformers for encrypted traffic classification

    Xinjie Lin, Gang Xiong, Gaopeng Gou, Zhen Li, Junzheng Shi, and Jing Yu. ET-BERT: A contextualized datagram representation with pre-training transformers for encrypted traffic classification. InProceedings of the ACM Web Conference, pages 633–642, 2022. 16

    2022

  5. [5]

    Intrusion detection using bidirectional LSTM recurrent neural network

    Hongyu Liu and Bo Lang. Intrusion detection using bidirectional LSTM recurrent neural network. InIT Professional, volume 21, pages 52–58. IEEE, 2019

    2019

  6. [6]

    Flow- bert: Learning network-flow representations for intrusion detection

    Gaetano Pellegrino, Christian Hammerschmidt, Radu State, and Thomas Engel. Flow- bert: Learning network-flow representations for intrusion detection. InIEEE International Symposium on Local and Metropolitan Area Networks (LANMAN), 2021

    2021

  7. [7]

    RWKV: Reinventing RNNs for the transformer era

    Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Bi- derman, et al. RWKV: Reinventing RNNs for the transformer era. InFindings of the Association for Computational Linguistics: EMNLP 2023, pages 14048–14064, 2023

    2023

  8. [8]

    Snort: Lightweight intrusion detection for networks

    Martin Roesch. Snort: Lightweight intrusion detection for networks. InUSENIX LISA, volume 99, pages 229–238, 1999

    1999

Show all 12 references
  1. [9]

    Ghorbani

    Iman Sharafaldin, Arash Habibi Lashkari, and Ali A. Ghorbani. Toward generating a new intrusion detection dataset and intrusion traffic characterization. InInternational Conference on Information Systems Security and Privacy (ICISSP), pages 108–116, 2018

    2018

  2. [10]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. InAdvances in Neural Information Processing Systems (NeurIPS), volume 30, 2017

    2017

  3. [11]

    Network traffic classification using convolutional neural networks.arXiv preprint arXiv:2006.09765, 2020

    Wei Wang et al. Network traffic classification using convolutional neural networks.arXiv preprint arXiv:2006.09765, 2020

    2006

  4. [12]

    End-to-end en- crypted traffic classification with one-dimensional convolution neural networks

    Wei Wang, Ming Zhu, Xuewen Zeng, Xiaozhou Ye, and Yiqiang Sheng. End-to-end en- crypted traffic classification with one-dimensional convolution neural networks. InIEEE International Conference on Intelligence and Security Informatics (ISI), pages 43–48, 2017. A Complete Evalua...

    2017

Pith tools

Reviewed June 28, 2026 · model on record in the stance chip above.