Pith. sign in

REVIEW 4 major objections 5 minor 24 references

Deciding When Not to Decide: Indeterminacy-Aware Intrusion Detection with NeutroSENSE

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper argues that an ensemble intrusion-detection system can abstain on ambiguous inputs by scoring its own disagreement, and that doing so catches most misclassifications before they trigger alerts.

desk verdict Entropy rebranded as neutrosophy, with an evaluation that doesn't support the headline claims; fixable, but not in current form. read the letter →

arxiv 2507.00003 v1 pith:AUQROQCO submitted 2025-06-05 cs.LG cs.AIcs.CRcs.NI

Eyhab Al-Masri This is my paper
classification cs.LGcs.AIcs.CRcs.NI
keywords neutrosophiclogicensembleclassificationindeterminacyintrusiondetectionabstentionuncertaintyquantificationIoTsecurityinterpretability
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes that an intrusion-detection system can decide when not to decide: by measuring how much an ensemble of classifiers disagrees with itself, it can flag uncertain predictions for human review instead of firing an alert. It defines an indeterminacy score I as the normalized entropy of the ensemble's averaged class probabilities and shows on an IoT attack dataset that misclassified samples score far higher (I≈0.62) than correct ones (I≈0.24). On confident samples (I≤0.4), accuracy rises to over 99.8%, at the cost of abstaining on roughly one-fifth of the traffic. The resulting framework, NeutroSENSE, is a step toward trust-aware, human-in-the-loop security at the network edge.

What carries the argument

The central object is the indeterminacy score I, defined as the normalized entropy of the ensemble's averaged class-probability vector, scaled to [0,1]. It is computed alongside truth T (confidence in the predicted class) and falsity F (aggregated confidence in all other classes). I carries the argument: high-I predictions are treated as cases where the ensemble lacks consensus, and the system abstains by sending them to a review queue. A companion adaptive mechanism sets a per-class 80th-percentile threshold of I so that inherently ambiguous classes such as Normal and DDoS are not over- or under-flagged.

What would settle it

Compute the area under the ROC curve of I as a binary predictor of correctness on a truly held-out, non-SMOTE test set; if the AUC approaches 0.5, indeterminacy carries no error signal and the abstention strategy would fail to catch misclassifications in practice.

Watch

Extended reading notes

Core claim

NeutroSENSE claims that decomposing an ensemble's averaged class probabilities into truth, falsity, and indeterminacy—with indeterminacy defined as normalized entropy—provides a reliable signal of when the classifier is likely to be wrong. On the IoT-CAD dataset, the ensemble of Random Forest, XGBoost, and Logistic Regression reaches 97% accuracy overall, but the average indeterminacy of errors (0.6223) is more than double that of correct predictions (0.2410). Flagging predictions above a global threshold (I>0.4) isolates over 12,000 ambiguous cases, many of them misclassifications, and a class-specific 80th-percentile threshold sends about 20% of each class's samples to a review queue while preserving accuracy on the rest. The paper concludes that indeterminacy is a strong proxy for epistemic uncertainty and that abstention based on it makes automated intrusion detection safer and more interpretable.

Load-bearing premise

The adaptive thresholds are computed from the test set's own ground-truth labels (the 80th percentile of I per class), so in a live edge deployment, where labels are unavailable, the model would have to guess those thresholds from unlabeled data, and the reported abstention behavior may change.

Editorial extensions

If this is right

  • Flagging high-indeterminacy predictions for human review can raise accuracy on retained samples to over 99.8% while abstaining on roughly 20% of traffic.
  • The clear separation between error indeterminacy (≈0.62) and correct indeterminacy (≈0.24) means a single scalar can prioritize which alerts need a human.
  • Class-specific thresholds let the system adapt to classes with inherently ambiguous decision boundaries, such as Normal and DDoS.
  • The framework is lightweight enough for edge gateways because it only requires the three base classifiers' probability outputs plus an entropy calculation.
  • The logging and review queue creates a feedback loop for retraining and threshold refinement over time.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct extension would be to test the same indeterminacy scoring on any ensemble of probabilistic classifiers and compare its error-prediction power against simpler baselines like plain entropy or margin, which the paper does not benchmark.
  • Because the adaptive thresholds are currently fit on labeled test data, a live deployment would need to estimate the 80th percentile from unlabeled streaming predictions or a separate validation set; whether the abstention behavior survives that shift remains an open question.
  • If indeterminacy tracks errors as cleanly as reported, it could also be used as a training signal—for example, to reweight or resample high-I regions—rather than only as a post-hoc abstention rule.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. NeutroSENSE is an ensemble of Random Forest, XGBoost, and Logistic Regression whose predictions are decomposed into truth, falsity, and indeterminacy scores, with indeterminacy defined as the normalized entropy of the ensemble-averaged class probabilities. The framework abstains from or flags for human review any prediction whose indeterminacy exceeds a global or class-specific threshold. On the IoT-CAD dataset, the paper reports 97% ensemble accuracy, an average indeterminacy of 0.62 for misclassified versus 0.24 for correct predictions, and 99.8% accuracy on low-indeterminacy retained samples. The paper concludes that indeterminacy is a strong proxy for epistemic uncertainty and that abstention improves trust in edge-deployed intrusion detection.

Significance. If the empirical claims were properly validated, the paper would offer a practical abstention mechanism for edge-based intrusion detection and a useful case study of uncertainty-aware IDS. The modular architecture is clearly described and the problem of 'knowing when not to decide' is relevant. The strengths of the paper are its clear system architecture and the identification of a real operational need. However, the evaluation protocol contains two design choices that contaminate the test data and in-sample threshold selection, and the paper does not compare against existing uncertainty baselines despite defining I as normalized entropy. No code or data splits are provided, so the quantitative results are not currently reproducible.

major comments (4)
  1. [IV.B / IV] The main empirical results are measured on a 'post-SMOTE stratified test set, drawn from the oversampled training data' (Section IV). Since SMOTE generates synthetic samples as convex interpolations of training points, this test set is not an independent sample from the original data distribution. Therefore the 97% accuracy in Table V, the I=0.6223 versus 0.2410 separation in Table VI, and the 99.8% confident-sample accuracy in Figure 5 are not estimates of generalization to the original data. The reserved 20% holdout set, which the paper itself says represents the pre-SMOTE distribution, is never used for the ensemble or neutrosophic scoring analyses; it should be the primary evaluation set.
  2. [IV.D and IV.E] The abstention thresholds are selected on the test set itself. Section IV.D sweeps I-thresholds on the test set and reports the accuracy at the resulting operating point, while Section IV.E states that 'we computed the 80th percentile indeterminacy score I within each class on the test set.' Per-class thresholds require ground-truth labels at evaluation time, so the reported accuracy-on-confident-samples is an in-sample fit rather than a prospective performance estimate. The paper should fix thresholds on a validation split and then evaluate on a clean, pre-SMOTE held-out set.
  3. [IV.F] The paper concedes that it did not benchmark against other uncertainty methods (e.g., entropy, margin). Because I is explicitly defined as normalized entropy of the ensemble-averaged class probabilities, an entropy-only abstention baseline under the same protocol is necessary to demonstrate that the neutrosophic decomposition contributes anything beyond standard confidence thresholding. Without such a baseline, the claimed advantage of NeutroSENSE over conventional uncertainty-based abstention is unsupported.
  4. [IV.D] The Youden Index is defined as Accuracy × Coverage, which deviates from the standard Youden's J statistical measure and is not justified. The paper also states that the Youden Index peaked at I≤0.8 while highlighting the I≤0.4 operating point as reaching 99.8% accuracy; coverage at that highlighted point is never reported, so the operational trade-off between accuracy and abstention coverage cannot be evaluated by the reader.
minor comments (5)
  1. [II.B/II.C] The headings of subsections II.B and II.C are both 'Uncertainty Modeling with Neutrosophic Logic'; the second heading should be changed (e.g., to 'Neutrosophic Logic for Intrusion Detection').
  2. [References [20]] Reference [20] is cited as 'Idhammad et al. (2023)' in the text and in Table I, but the bibliography lists the publication year as 2018; please reconcile the citation.
  3. [III.B / IV.C] The formal definitions of T, F, and I are not given; please provide explicit equations, including the normalization constant for the entropy-based I, so that the scoring is reproducible.
  4. [Abstract] The abstract states 'achieved 97% accuracy' without qualifier; this number comes from the post-SMOTE test set and should be identified as such to avoid misleading readers.
  5. [IV.D / Figure 5] Coverage at the highlighted I≤0.4 operating point is not reported; the text or figure should state what fraction of samples is retained at each threshold so that the accuracy-coverage trade-off is transparent.

Circularity Check

2 steps flagged · score 6.0 of 10

Abstention benefit is fitted to the test set and evaluated on a SMOTE-derived test set, so the reported I-error separation is partly an in-sample artifact; the I-score definition itself is not circular.

  1. fitted input called prediction [Section IV.D (Threshold Sweep and Youden Index) and Section IV.E (Adaptive Abstention & Flagging)]
    "To implement this, we computed the 80th percentile indeterminacy score I within each class on the test set. ... At I≤0.9, the ensemble achieves 97.31% accuracy with full coverage ... filtering at I≤0.4 yields even higher precision, reaching over 99.8% accuracy on confident samples."

    The abstention thresholds are computed from the same test set that is then used to report the accuracy gain: per-class 80th-percentile thresholds are derived from test-set indeterminacy scores and labels, and the global threshold is selected by a Youden-index sweep over test predictions. Therefore the reported 99.8% accuracy on confident samples and the I=0.62 vs. 0.24 separation are in-sample fitted values, not out-of-sample predictions of an independently specified abstention rule. The validation of 'indeterminacy as a proxy for uncertainty' is fitted to the evaluation data by construction.

  2. other [Section IV (Evaluation setup, test sets)]
    "All models were evaluated on two distinct test sets: (1) a held-out validation set representing the original data distribution prior to SMOTE balancing ... and (2) a post-SMOTE stratified test set, drawn from the oversampled training data, used for deeper analysis of ensemble performance and neutrosophic scoring."

    The main neutrosophic-scoring results and the I-vs-error analysis are produced on a 'test set' drawn from post-SMOTE oversampled training data, which consists of original training samples plus synthetic neighbors generated from them. Evaluating the claimed indeterminacy-error correlation and abstention accuracy on this set measures in-sample confidence and memorization rather than generalization to real unseen traffic. The paper concedes no entropy or margin baselines were run, so the reported abstention behavior is not established as an independent, predictive result.

full rationale

The definition of I as normalized entropy is not circular, and the qualitative claim that misclassified samples tend to have higher entropy is an empirical hypothesis that could be tested properly. However, the paper's central quantitative evidence is compromised in a circular way: the adaptive thresholds are computed from the test set's own indeterminacy distribution and labels, and the global threshold is selected by sweeping I and maximizing Youden index on the same test set. Reporting accuracy under those fitted thresholds as 'abstention improves accuracy' is a post-hoc selection, not a prediction. In addition, the test set used for the neutrosophic scoring is drawn from SMOTE-oversampled training data, so the evaluation includes synthetic near-duplicates of training samples rather than a truly held-out sample. There is no load-bearing self-citation chain, and the I-score itself is not derived from the labels. The score is 6 rather than higher because the relationship is not forced by definition; a proper held-out pre-SMOTE evaluation and label-free threshold selection could support the claim independently.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new mathematical entities; T, F, and I are renamings of the predicted-class probability, the sum of other-class probabilities, and normalized entropy. The free parameters are all thresholds fitted to the test set, and the key assumptions concern the validity of averaging uncalibrated model probabilities, the representativeness of the SMOTE-contaminated test set, and the operational suitability of entropy as an abstention signal.

free parameters (3)
  • Global indeterminacy threshold for flagging = 0.4
    Used to define high-indeterminacy samples in Section IV.C. Chosen from the test-set distribution, not from a separate validation procedure.
  • Per-class 80th percentile indeterminacy thresholds = One threshold per class, computed as the 80th percentile of I on the test set
    Section IV.E computes these thresholds on the test set using ground-truth labels, which requires test labels at deployment and tunes the abstention rule to the evaluation data.
  • Youden-index threshold = I <= 0.8
    Section IV.D reports the Youden Index peaked at I <= 0.8. This is selected from a threshold sweep on the test set, making the reported accuracy-coverage trade-off a fitted result.
assumptions (3)
  • domain assumption Averaged softmax probabilities from Random Forest, XGBoost, and Logistic Regression are comparable and can be meaningfully combined as a consensus distribution.
    Section III.B.3 and Section IV.B combine class probabilities from the three models by averaging. Tree-based models, especially Random Forest, are known to produce poorly calibrated probabilities, so the averaging may not yield a well-calibrated consensus.
  • domain assumption The post-SMOTE test set, drawn from oversampled training data, is representative of real-world IoT traffic.
    Section IV states the second test set is "drawn from the oversampled training data." This assumes synthetic SMOTE samples preserve the real distribution, which is questionable and is a core weakness of the evaluation.
  • ad hoc to paper High indeterminacy, defined as normalized entropy, is an appropriate operational proxy for epistemic uncertainty in this deployment context.
    The paper postulates this proxy in Section IV.C without comparing it to other uncertainty measures such as margin, variance, or calibration-based scores, and without a deployment-time method for setting thresholds.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Deciding When Not to Decide: Indeterminacy-Aware Intrusion Detection with NeutroSENSE." pith.science (2026). https://pith.science/paper/AUQROQCO

@misc{pith2026250700003,
  author       = {Pith},
  title        = {Pith review of: Deciding When Not to Decide: Indeterminacy-Aware Intrusion Detection with NeutroSENSE},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AUQROQCO}},
  note         = {Machine review of arXiv:2507.00003}
}
read the original abstract

This paper presents NeutroSENSE, a neutrosophic-enhanced ensemble framework for interpretable intrusion detection in IoT environments. By integrating Random Forest, XGBoost, and Logistic Regression with neutrosophic logic, the system decomposes prediction confidence into truth (T), falsity (F), and indeterminacy (I) components, enabling uncertainty quantification and abstention. Predictions with high indeterminacy are flagged for review using both global and adaptive, class-specific thresholds. Evaluated on the IoT-CAD dataset, NeutroSENSE achieved 97% accuracy, while demonstrating that misclassified samples exhibit significantly higher indeterminacy (I = 0.62) than correct ones (I = 0.24). The use of indeterminacy as a proxy for uncertainty enables informed abstention and targeted review-particularly valuable in edge deployments. Figures and tables validate the correlation between I-scores and error likelihood, supporting more trustworthy, human-in-the-loop AI decisions. This work shows that neutrosophic logic enhances both accuracy and explainability, providing a practical foundation for trust-aware AI in edge and fog-based IoT security systems.

Figures

Figures reproduced from arXiv: 2507.00003 by the authors.

Figure 1
Figure 1. Architecture of the Proposed NeutroSENSE Framework with Neutrosophic Reasoning and Adaptive Abstention [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 3
Figure 3. While most predictions exhibit low to moderate [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 2
Figure 2. Confusion matrix for XGBoost predictions on the holdout test set [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figures from the paper (2 more)
Figure 6
Figure 6. Figure 6: Indeterminacy scores by class. Normal and DDoS show higher uncertainty, supporting class-aware abstention [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: Indeterminacy scores by prediction correctness. Higher uncertainty in errors supports abstention strategies [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

24 extracted references · 22 canonical work pages

  1. [1]

    Defending the digital Frontier: IDPS and the battle against Cyber threat

    Azam, Hamza, Mohammad Irfan Dulloo, Muhammad Hassan Majeed, Janelle Phang Hui Wan, et al., "Defending the digital Frontier: IDPS and the battle against Cyber threat." International Journal of Emerging Multidisciplinaries Computer Science & AI, 2, no. 1 (2023): 253

    work page 2023

  2. [2]

    Learning Tec hniques-Based Malware Detection: A Comprehensive Review

    Ali, Sarah Fouad, Musaab Riyadh Abdulrazzaq, and Methaq Talib Gaata. "Learning Tec hniques-Based Malware Detection: A Comprehensive Review." Mesopotamian J. of CyberSecurity 5, no. 1 (2025): 273-300

    work page 2025

  3. [3]

    A survey of data mining and machine learning methods for cyber security intrusion detection

    Buczak, Anna L., and Erhan Guven. "A survey of data mining and machine learning methods for cyber security intrusion detection." IEEE Communications surveys & tutorials 18, no. 2 (2015): 1153-1176

    work page 2015

  4. [4]

    Outside the closed world: On using machine learning for network intrusion detection

    Sommer, Robin, and Vern Paxson. "Outside the closed world: On using machine learning for network intrusion detection." In 2010 IEEE symposium on security and privacy, pp. 305-316. IEEE, 2010

    work page 2010

  5. [5]

    Concrete problems in AI safety

    Amodei, Dario, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. "Concrete problems in AI safety." arXiv preprint arXiv:1606.06565 (2016)

    arXiv 2016

  6. [6]

    Survey of intrusion detection systems: techniques, datasets and challenges

    Khraisat, Ansam, Iqbal Gondal, Peter Vamplew, and Joarder Kamruzzaman. "Survey of intrusion detection systems: techniques, datasets and challenges." Cybersecurity 2, no. 1 (2019): 1-22

    work page 2019

  7. [7]

    Towa rds a rigorous science of interpretable machine learning

    Doshi-Velez, Finale, and Been Kim. "Towa rds a rigorous science of interpretable machine learning." arXiv preprint arXiv:1702.08608 (2017)

    arXiv 2017

  8. [8]

    " Why should i trust you?

    Ribeiro, Marco Tulio, Sameer Singh, and Carlos Guestrin. "" Why should i trust you?" Explaining the predictions of any classifier." In Proceedings of the 22n d ACM SIGKDD international conference on knowledge discovery and data mining, pp. 1135-1144. 2016

    work page 2016

Show all 24 references
  1. [9]

    Ensemble methods in machine learning

    Dietterich, Thomas G. "Ensemble methods in machine learning." In International workshop on multiple classifier systems, pp. 1 -15. Berlin, Heidelberg: Springer Berlin Heidelberg, 2000

    2000

  2. [10]

    A unifying field in Logics: Neutrosophic Logic

    Smarandache, Florentin. "A unifying field in Logics: Neutrosophic Logic." In Philosophy, pp. 1-141. American Research Press, 1999

    1999

  3. [11]

    A deep learning approach for image and te xt classification using neutrosophy

    Wajid, Mohd Anas, Aasim Zafar, and Mohammad Saif Wajid. "A deep learning approach for image and te xt classification using neutrosophy." International Journal of Info. Technology 16, no. 2 (2024): 853-859

    2024

  4. [12]

    Ensemble neural networks using interval neutrosophic sets and bagging

    Kraipeerapun, Pawalai, Chun Che Fung, and Kok Wai Wong. "Ensemble neural networks using interval neutrosophic sets and bagging." Int ’l Conference on Natural Computation, vol. 1, pp. 386-390. IEEE, 2007

    2007

  5. [13]

    An ensemble design of intrusion detection system for handling uncertainty using Neutrosophic Logic Classifier

    Kavitha, B., S. Karthikeyan, and P. Sheeba Maybell. "An ensemble design of intrusion detection system for handling uncertainty using Neutrosophic Logic Classifier." Knowledge-Based Systems 28 (2012): 88-96

    2012

  6. [14]

    Simple and scalable predictive uncertainty estimation using deep ensembles

    Lakshminarayanan, Balaji, Alexander Pritzel, and Charles Blundell. "Simple and scalable predictive uncertainty estimation using deep ensembles." Advances in neural inf. processing systems 30 (2017)

    2017

  7. [15]

    A novel neutrosophic weighted extreme learning machine for imbalanced data set

    Akbulut, Yaman, Abdulkadir Şengür, Yan hui Guo, and Florentin Smarandache. "A novel neutrosophic weighted extreme learning machine for imbalanced data set." Symmetry 9, no. 8 (2017): 142

    2017

  8. [16]

    IoT-CAD: A comprehensive Digital Forensics dataset for AI- based Cyberattack Attribution Detection methods in IoT environments

    Mohamed, Hania, Nickolaos Koroniotis, Francesco Schiliro, and Nour Moustafa. "IoT-CAD: A comprehensive Digital Forensics dataset for AI- based Cyberattack Attribution Detection methods in IoT environments." Ad Hoc Networks (2025): 103840

    2025

  9. [17]

    Machine learning in neutrosophic environment: A survey

    Elhassouny, Azeddine, Soufiane Idbrahim, and Florentin Smarandache. "Machine learning in neutrosophic environment: A survey." Infinite study (2019)

    2019

  10. [18]

    Feature deduction and ensemble design of intrusion detection systems

    Chebrolu, Srilatha, Ajith Abraham, and Johnson P. Thomas. "Feature deduction and ensemble design of intrusion detection systems." Computers & security 24, no. 4 (2005): 295-307

    2005

  11. [19]

    A new approach to intrusion detection using Artificial Neural Networks and fuzzy clustering

    Wang, Gang, Jinxing Hao, Jian Ma, and Lihua Huang. "A new approach to intrusion detection using Artificial Neural Networks and fuzzy clustering." Expert systems with app., 37, no. 9 (2010): 6225-6232

    2010

  12. [20]

    Detection system of HTTP DDoS attacks in a cloud envir onment based on information theoretic entropy and random forest

    Idhammad, Mohamed, Karim Afdel, and Mustapha Belouch. "Detection system of HTTP DDoS attacks in a cloud envir onment based on information theoretic entropy and random forest." Security and Communication Networks 2018, no. 1 (2018): 1263123

    2018

  13. [21]

    Search: A collab orative and intelligent nids architecture for sdn -based cloud iot networks

    Nguyen, Tri Gia, Trung V. Phan, Binh T. Nguyen, Chakchai So-In, Zubair Ahmed Baig, and Surasak Sanguanpong. "Search: A collab orative and intelligent nids architecture for sdn -based cloud iot networks." IEEE access 7 (2019): 107678-107694

    2019

  14. [22]

    Binary classification using ensemble neural networks and interval neutrosophic sets

    Kraipeerapun, Pawalai, and Chun Che Fung. "Binary classification using ensemble neural networks and interval neutrosophic sets." Neurocomputing 72, no. 13-15 (2009): 2845-2856

    2009

  15. [23]

    Intrusion detection system: A comprehensive review

    Liao, Hung -Jen, Chun-Hung Richard Lin, Ying -Chih Lin, and Kuang - Yuan Tung. "Intrusion detection system: A comprehensive review." Journal of network and computer applications 36, no. 1 (2013): 16-24

    2013

  16. [24]

    SMOTE: synthetic minority over -sampling technique

    Chawla, Nitesh V., Kevin W. Bowyer, Lawrence O. Hall, and W. Philip Kegelmeyer. "SMOTE: synthetic minority over -sampling technique." Journal of artificial intelligence research 16 (2002): 321-357. Fig. 7. Indeterminacy scores by prediction correctness. Higher uncertainty in e...

    2002

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.