Pith. sign in

REVIEW 4 major objections 5 minor 16 references

Dueling Deep Q-Learning for Intrusion Detection

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A dueling deep Q-network trained with reward-based learning reports 99.68% average accuracy in classifying CIC-IDS2018 network attacks, above earlier DQN-based intrusion detection systems.

desk verdict A cleanly-written engineering paper whose headline accuracy is in-sample; the evaluation protocol undercuts the main claim. read the letter →

arxiv 2608.11291 v1 pith:CA35TMSW submitted 2026-08-11 cs.CR cs.LG

Logan Luna (Georgia Institute of Technology) , Matthew P. Berkowitz (Embry-Riddle Aeronautical University) , Laxima Niure Kandel (Embry-Riddle Aeronautical University) , Sirio Jansen-S'anchez (Embry-Riddle Aeronautical University) This is my paper · ORCID
classification cs.CRcs.LG
keywords intrusiondetectionreinforcementlearningduelingdeepQ-networkCIC-IDS2018SHAPcybersecuritythreathuntingmulticlassclassification
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that a reward-based dueling deep Q-network can serve as an intrusion detection classifier, and reports that it reaches 99.68% average accuracy across the attack types in the CIC-IDS2018 dataset. The authors frame this as an improvement over prior Q-learning-based intrusion detection work, which reached about 88% accuracy, and as competitive with supervised models such as random forest. The paper also integrates SHAP to identify which flow features drive the model's decisions. The authors explicitly note that the model was trained and evaluated in the same environment, so the 99.68% figure is an in-sample report; the claimed adaptability to new attack types is not tested in the experiments described.

What carries the argument

The load-bearing object is the dueling deep Q-network, which decomposes the action-value function as $$Q(s,a)=V(s)+\left(A(s,a)-\frac{1}{|A|}\sum_{a'}A(s,a')\right)$$ so that one stream learns the value of a state $V(s)$ and another learns the relative advantage $A(s,a)$ of each action. The two streams share two hidden layers of 128 and 64 units with batch normalization and ReLU activations, then split into a value head and an advantage head. The environment treats each network flow as a state, and the reward is $r_t=+1\cdot S_l\cdot C_a+\min(0.5\log(\text{streak}),2.0)$ for a correct action and $-1\cdot S_l\cdot C_a$ for an incorrect one, where $S_l$ is a severity weight and $C_a=0.5+\text{confidence}/2$ scales with the network's confidence. The dueling decomposition is what the paper credits for stable and efficient learning; the shaped reward is what turns classification into a Q-learning problem.

What would settle it

Retrain the model on a proper training split of CIC-IDS2018 and evaluate on a held-out split, such as a different week of traffic, then compare that accuracy with the reported 99.68%; also recompute the average with the 173 web attack samples included, since the reported number excludes the one class where the model scores zero.

Watch

Extended reading notes

Core claim

The paper's central claim is that a dueling deep Q-network, trained with a reward function built from true labels, classifies network flows in the CIC-IDS2018 dataset with a weighted average accuracy of 99.68%, with per-class F1 scores above 0.99 for the five classes with enough samples; the web attack class, with only 173 samples, is excluded from the main analysis. The authors attribute the improvement over earlier DQN-based IDS work to the dueling architecture's separate value and advantage streams and to the larger training set of 2,177,804 samples. SHAP analysis shows that decisions rest on features such as RST flag count, PSH flag count, backward packet length, initial window bytes, ACK flag count, and byte and packet rates.

Load-bearing premise

The load-bearing premise is that accuracy measured in the same environment used for training is a valid report of detection performance; the paper states this was the setup and no held-out evaluation is described.

Editorial extensions

If this is right

  • A reward-based dueling DQN can match the accuracy of the best supervised models on CIC-IDS2018, making reinforcement learning a viable alternative for multiclass intrusion detection.
  • Because the reward uses per-class severity weights, the same architecture can be tuned to penalize missed high-severity attacks more heavily without changing the network structure.
  • The SHAP feature attributions provide a feature-level account of each classification, which could make RL-based detections more auditable when deployed.
  • Training on 2.18 million samples shows the approach scales to large datasets, though the authors acknowledge added computational cost over supervised classifiers.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the paper reports no held-out test set, the 99.68% figure is best interpreted as training-environment accuracy; any statement about detecting unseen traffic needs a separate evaluation split.
  • The reward is assigned from the true label of each flow, so the model is functionally a supervised classifier with a shaped reward; the paper's claimed advantage in adapting to novel attacks is not demonstrated by the reported experiments.
  • The streak bonus in the reward encourages the agent to keep outputting its current prediction, which could inflate accuracy on long contiguous runs of similar traffic; shuffled or stream-based evaluation would reveal how much this mechanism contributes.
  • A direct ablation comparing dueling DQN with plain DQN and with a supervised MLP on identical data would isolate whether the dueling structure or the larger sample count explains the jump from the 88% baseline.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a dueling deep Q-network (Dueling DQN) for network intrusion detection, trained on the CIC-IDS2018 dataset, and reports an average accuracy of 99.68% across multiple attack types. The authors also integrate SHAP for explainability. The main claim is that this reward-based, dueling Q-learning approach outperforms prior RL-based IDS methods. However, the evaluation is performed in the same environment used for training, as acknowledged in Section V, and no held-out test set or cross-validation is described. The Web attack class is excluded post hoc after showing zero performance, and the reward function is based on the true class label, making the approach functionally similar to supervised classification.

Significance. If the claimed performance were supported by a rigorous evaluation, the paper could make a useful contribution to RL-based intrusion detection by demonstrating the dueling architecture's effectiveness on a modern benchmark. The SHAP analysis also provides interpretability, which is valuable. However, the central accuracy claim is currently unsupported because the evaluation is in-sample. The paper ships no code and reports no error bars, and the comparison to prior work is confounded by differing datasets and protocols. As presented, the manuscript does not establish its main contribution.

major comments (4)
  1. [Section V, 'Deployment in Real-World Scenarios'] The paper explicitly states that the model was trained and evaluated in the same environment, and no train/test split, temporal partition, or cross-validation is described anywhere in Sections III or IV. The 99.68% accuracy reported in Table V and the Abstract is therefore an in-sample fit, and the comparisons in Table IV are not valid evidence of generalization. This is load-bearing because every performance claim rests on this number. The authors must re-run the experiments with a held-out test set (ideally a temporal split, given the sequential nature of the data) and report error bars or confidence intervals.
  2. [Section IV.A, Table V] The Web attack class is excluded from the reported average after observing its performance, which is exactly zero (precision, recall, and F1-score are 0.0000). This post-hoc exclusion changes the class set and inflates the reported average accuracy. The authors should either include the Web attack class with appropriate class-balancing techniques or report results for all classes without selective removal, and justify any exclusion a priori.
  3. [Section III.B, Reward Calculation] The reward function depends directly on the true label l_t in both the correct and incorrect branches. Consequently, the learning signal is the ground-truth class label, and the agent is effectively performing supervised classification with a shaped reward rather than learning from environment feedback. This undercuts the paper's framing as a reinforcement learning approach that can adapt to new attack types. The authors should clarify how this differs from a standard classifier and provide evidence of adaptation to unseen attacks, such as evaluating on attack types not seen during training.
  4. [Table IV] The comparison with prior work is not meaningful because the studies use different datasets (e.g., NSL-KDD vs. CIC-IDS2018), different class sets, and likely different evaluation protocols. Table IV cannot support the claim of state-of-the-art performance. The authors should compare on the same benchmark with identical train/test partitions, or explicitly discuss the limitations of cross-study comparisons.
minor comments (5)
  1. [Section V] The sentence 'This model was trained and evaluated in the same environment it was rained in' contains a typo: 'rained' should be 'trained'.
  2. [Figure 5 caption] The caption reads 'Explainability resutls for DQN Decision Making'; 'resutls' should be 'results'.
  3. [Table III] The paper mentions that there are over 80 features, but Table III lists only seven. Please clarify the full feature set used by the model and whether feature selection was applied.
  4. [Algorithm 1] The diagram in Algorithm 1 contains labels '1' and '5' that are ambiguous; please provide a clear legend or textual description of the architecture dimensions.
  5. [References] Reference [9] appears to have an incomplete or incorrectly formatted DOI (10.1109/ICCS.2023.983451); please verify and correct the citation details.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation found: the reported 99.68% accuracy is an in-sample measurement, not a quantity constructed from the reward function or from a self-citation chain.

full rationale

After walking the paper's derivation chain, I find no circular step under the definitions used here. The formal content is a standard dueling DQN update (Eqs. 1-2) plus a custom Gym environment whose reward (Section III.B) is a function of the ground-truth label l_t. The reported 99.68% accuracy (Table V) is an empirical measurement on the training environment, not a quantity obtained by substituting the reward definition into an equation and recovering the claim. Section V explicitly concedes: 'This model was trained and evaluated in the same environment it was rained in.' That concession undermines any generalization claim, but an in-sample accuracy report is a validity and soundness issue, not a circular reduction. There is no self-citation chain, no imported uniqueness theorem, and no ansatz smuggled via prior work. The motivation in Section I.B that RL 'learns from a reward structure rather than being based on purely labeled data' is overstated because the reward uses labels, but the paper's accuracy claim does not reduce to the reward definition by construction. Therefore the score is 0 rather than a positive circularity score.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central result rests on roughly a dozen hand-chosen hyperparameters, two of which (reward severity weights and shaping constants) are not even reported. The modeling axioms include treating classification as an MDP and trusting the benchmark labels. No new physical or conceptual entities are introduced.

free parameters (5)
  • Learning rate = 0.001
    Set in Table II; chosen by hand, affects convergence and final policy.
  • Discount factor gamma = 0.99
    Set in Table II; with immediate per-flow rewards, gamma is an arbitrary bootstrapping weight rather than a meaningful temporal discount.
  • Epsilon schedule = 1.0 start, 0.1 end, 0.999 decay
    Exploration schedule in Table II; chosen by hand, controls the balance between exploration and exploitation during training.
  • Reward severity weights S_l = Unspecified
    The reward function in Section III.B depends on severity weights per class, but the paper never lists their values; changing them would change the learned policy and the reported per-class performance.
  • Reward shaping constants = Ca = 0.5 + confidence/2; streak cap 2.0
    Arbitrary constants in the reward formula that shape the learning signal; no sensitivity analysis is provided.
assumptions (4)
  • domain assumption Each network flow is an independent state; the environment is stationary and Markovian.
    Section III.B processes flows sequentially but treats each flow as an individual state, ignoring temporal dependencies between flows that matter for detecting multi-stage attacks.
  • domain assumption CIC-IDS2018 labels are correct and the dataset distribution represents real-world intrusion scenarios.
    Section II.C relies on the benchmark as ground truth; label noise or distribution shift would change all reported metrics.
  • ad hoc to paper Maximizing label-match reward under this MDP is a valid training objective for an intrusion detector.
    The reward function is defined directly on the true class label, making the RL objective a reparameterized supervised classification loss; the paper presents this as principled RL without justifying the MDP formulation.
  • ad hoc to paper In-sample evaluation on the training environment is acceptable evidence of model performance.
    The limitations admit training and evaluation in the same environment, but the Results section still reports the accuracy as the headline performance without a caveat.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Dueling Deep Q-Learning for Intrusion Detection." pith.science (2026). https://pith.science/paper/CA35TMSW

@misc{pith2026260811291,
  author       = {Pith},
  title        = {Pith review of: Dueling Deep Q-Learning for Intrusion Detection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CA35TMSW}},
  note         = {Machine review of arXiv:2608.11291}
}
read the original abstract

Intrusion detection systems (IDS) and automated systems for detecting and reporting cyber threats, are commonly handled via supervised machine learning methods. Though effective, these models struggle to effectively adapt to new attack types. This study proposes a novel approach by employing a reward-based, dueling Q-learning model for IDS, achieving an average accuracy of 99.68% across multiple attack classes. The proposed model has a dueling network architecture which separates its predictions into value and advantage streams. This has the benefit of improving learning efficiency and stability. The model was trained on the CIC-IDS2018, a benchmark dataset based on real-world intrusion detection scenarios, having multiple attack classes such as DDoS, botnets, and brute-force attacks. Furthermore, Explainable AI (XAI), specifically SHAP (SHapley Additive exPlanations), was also integrated into the training and evaluation process to provide interpretability into the model's predictions.

Figures

Figures reproduced from arXiv: 2608.11291 by the authors.

Figure 1
Figure 1. RL architecture visualization with input, hidden, and [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Architecture of Dueling DQN (shared layers feeding [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. A single layer’s forward pass: input vector [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Confusion Matrix for DQN Agent The proposed hierarchical Q-learning framework has displayed notable improvements over similar previous studies that sought to implement RL for IDSs as shown in Table IV. While Alavizadeh et al. [8] achieved an accuracy of 88%, our DQN at…
Figure 5
Figure 5. Figure 5: Explainability resutls for DQN Decision Making (a) RL [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

16 extracted references · 14 canonical work pages

  1. [8]

    Deep q-learning based reinforcement learning approach for network intrusion detection.Computers, 11(3):41, 2022

    Hooman Alavizadeh, Julian Jang-Jaccard, and Hootan Alavizadeh. Deep q-learning based reinforcement learning approach for network intrusion detection.Computers, 11(3):41, 2022. doi: 10.3390/computers11030041. URL https://www.mdpi.com/2073-431X/11/3/41

    doi:10.3390/computers11030041 2022

  2. [1]

    Khraisat, I

    A. Khraisat, I. Gondal, P. Vamplew, and J. Kamruzzaman. Survey of intrusion detection systems: techniques, datasets and challenges.Cyber- security, 2(1):1–22, 2019

    work page 2019

  3. [2]

    Garc ´ıa-Teodoro, J

    P. Garc ´ıa-Teodoro, J. D´ıaz-Verdejo, G. Maci´a-Fern´andez, and E. V´azquez. Anomaly-based network intrusion detection: Techniques, systems and challenges.Computers & Security, 28(1-2):18–28, 2009

    work page 2009

  4. [3]

    Outside the closed world: On using machine learning for network intrusion detection

    Robin Sommer and Vern Paxson. Outside the closed world: On using machine learning for network intrusion detection. InIEEE Symposium on Security and Privacy. IEEE, 2010

    work page 2010

  5. [4]

    Anomaly detec- tion: A survey.ACM Computing Surveys (CSUR), 41(3):1–58, 2009

    Varun Chandola, Arindam Banerjee, and Vipin Kumar. Anomaly detec- tion: A survey.ACM Computing Surveys (CSUR), 41(3):1–58, 2009

    work page 2009

  6. [5]

    J. M. Est ´evez-Tapiador, P. Garc´ıa-Teodoro, and J. E. D´ıaz-Verdejo. Tech- niques for intrusion detection in computer networks.ACM Computing Surveys (CSUR), 36(3):235–276, 2004

    work page 2004

  7. [6]

    Data breach investigations report, 2022

    Verizon. Data breach investigations report, 2022. Retrieved from https: //www.verizon.com/business/resources/reports/dbir/

    work page 2022

  8. [7]

    Sutton and Andrew G

    Richard S. Sutton and Andrew G. Barto.Reinforcement Learning: An Introduction. MIT Press, Cambridge, MA, USA, 1st edition, 1998. ISBN 978-0262193986

    work page 1998

Show all 16 references
  1. [9]

    Hierarchical multi- agent reinforcement learning for cyber network defense.Proceedings of the 2023 International Conference on Cybersecurity, 2023

    Aditya Vikram Singh, Ethan Rathbun, Emma Graham, Lisa Oakley, Simona Boboila, Alina Oprea, and Peter Chin. Hierarchical multi- agent reinforcement learning for cyber network defense.Proceedings of the 2023 International Conference on Cybersecurity, 2023. doi: 10.1109/ICCS.2023...

    2023

  2. [10]

    Op- timizing intrusion detection systems in three phases on the cse-cic- ids-2018 dataset.MDPI Computers, 12(12):245, 2023

    Surasit Songma, Theera Sathuphanand, and Thanakorn Pamutha. Op- timizing intrusion detection systems in three phases on the cse-cic- ids-2018 dataset.MDPI Computers, 12(12):245, 2023. doi: 10.3390/ computers1212245. URL https://www.mdpi.com/2073-431X/12/12/245

    2018

  3. [11]

    Human-level control through deep reinforcement learning.Nature, 518(7540):529–533, 2015

    V olodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning.Nature, 518(7540):529–533, 2015. doi: 10.1038/ n...

    2015

  4. [12]

    Dueling network architectures for deep reinforce- ment learning.Proceedings of the 33rd International Conference on Machine Learning (ICML), 2016

    Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Van Hasselt, Marc Lanctot, and Nando De Freitas. Dueling network architectures for deep reinforce- ment learning.Proceedings of the 33rd International Conference on Machine Learning (ICML), 2016. URL https://arxiv.org/abs/1602.05110

    2016 arXiv

  5. [13]

    Sharafaldin, A

    I. Sharafaldin, A. H. Lashkari, and A. A. Ghorbani. Toward generating a new intrusion detection dataset and intrusion traffic characterization. In Proceedings of the 5th International Conference on Information Systems Security and Privacy (ICISSP), pages 108–116. SciTePress, 2...

    2018

  6. [14]

    Intrusion detection system development using tree-based machine learning algorithms.International Journal of Computer Networks & Communications, 15(4):73–85, 2023

    Witcha Chimphlee and Siriporn Chimphlee. Intrusion detection system development using tree-based machine learning algorithms.International Journal of Computer Networks & Communications, 15(4):73–85, 2023. URL https://aircconline.com/ijcnc/V15N4/15423cnc06.pdf

    2023

  7. [15]

    Hast-ids: Learning hierarchical spatial-temporal features using deep neural networks to improve intrusion detection.IEEE Access, 6:19174–19184, 2018

    Xin Zhao, Xiaoling Hu, and Wenjing Chen. Hast-ids: Learning hierarchical spatial-temporal features using deep neural networks to improve intrusion detection.IEEE Access, 6:19174–19184, 2018. doi: 10.1109/ACCESS.2018.2811762. URL https://ieeexplore.ieee.org/ document/8171733

    2018

  8. [16]

    Beechey, T

    D. Beechey, T. M. S. Smith, and ¨O. S ¸ims ¸ek. Explaining reinforcement learning with shapley values. InProceedings of the 40th Interna- tional Conference on Machine Learning, volume 202 ofProceedings of Machine Learning Research, pages 2003–2014. PMLR, 2023. URL https://proc...

    2003

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.