The reviewed record of science sign in
Pith

arxiv: 2606.11565 · v1 · pith:K3NDZWHJ · submitted 2026-06-10 · cs.CR

A Deterministic Forensic Preprocessing Framework for Heterogeneous Network Datasets: Formal Foundations, Implementation, and Empirical Validation

Ravi Chaudhary , Reza Ryan , Nasim Ferdosian , Nickson M. Karie , Qian Li This is my paper

Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel 2026-06-27 09:38 UTCgrok-4.3pith:K3NDZWHJrecord.jsonopen to challenge →

classification cs.CR
keywords forensic preprocessingheterogeneous network datasetsdeterministic frameworkschema normalisationtemporal normalisationprovenance trackingreproducibilityintrusion detection systems
0
0 comments X

The pith

A set-theoretic framework converts heterogeneous network datasets into a reproducible canonical form that preserves all information and tracks provenance.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces transformations that standardize incompatible schemas, timestamps, and origins from network logs into one consistent output. These steps rely on set-theoretic definitions and are backed by four theorems that prove the process is always deterministic, loses no data, and records full provenance. A chunk-based design keeps memory use bounded regardless of input size. Tests on three public datasets confirm every run produces exactly the same result. This addresses gaps in current ad-hoc methods that can produce varying outputs and undermine evidence reliability.

Core claim

By defining schema normalisation to unify field structures, temporal normalisation to align all timestamp formats, and provenance tracking to record data origins using set theory, the framework produces a canonical dataset form where repeated application on the same input always yields identical output, all original information remains recoverable, and complete provenance is maintained, as shown by four theorems and 100 percent consistency on UNSW-NB15, IoT-23, and TON_IoT.

What carries the argument

Three set-theoretic preprocessing transformations (schema normalisation, temporal normalisation, provenance tracking) plus four supporting theorems on determinism, preservation, and completeness, realised in a chunk-based processing architecture.

If this is right

  • Repeated runs on the same heterogeneous input always generate identical canonical output.
  • Every original record and field remains recoverable from the transformed dataset.
  • Timestamp alignment works across all formats present in the tested intrusion detection and IoT collections.
  • Memory use stays bounded by chunk size even as record counts reach hundreds of millions.
  • Provenance records capture the full history of each transformed element.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same normalisation steps could be applied to other forensic data types such as packet captures or system logs if their schemas are first expressed in set form.
  • Adoption would allow automated tools to compare preprocessed evidence across separate investigations without format mismatches.
  • The theorems could be extended to prove additional properties such as order preservation under specific merge operations.
  • Empirical checks on datasets containing more conflicting timestamp encodings would test whether the temporal normalisation theorem holds beyond the three collections already used.

Load-bearing premise

The mathematical definitions of the three transformations are enough to resolve every schema, timestamp, and origin mismatch in real network datasets without dropping forensic value or creating fresh inconsistencies.

What would settle it

Execute the framework twice on identical input from any of the three evaluated datasets and check whether any field value, record order, or timestamp differs between the two outputs.

Figures

Figures reproduced from arXiv: 2606.11565 by Nasim Ferdosian, Nickson M. Karie, Qian Li, Ravi Chaudhary, Reza Ryan.

Figure 1
Figure 1. Figure 1: Deterministic forensic preprocessing framework architecture. The primary contributions of this work are summarised as follows [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Set-theoretic schema canonicalisation. Definition 3 (Canonicalization): Given a dataset D = (R,S)and schema mapping ϕ:S → (Sc ∪ {⊥m}), the canonicalised dataset retains: - all canonical fields sc ∈ Sc such that ∃ s ∈ Swith ϕ(s) = sc , - all unmapped fields s ∈ S where ϕ(s) =⊥m, preserved unchanged. Canonicalization ensures that all datasets follow a consistent structure. Fields that can be standardized are… view at source ↗
Figure 3
Figure 3. Figure 3: Deterministic temporal normalisation workflow. Algorithm 2: Temporal Normalization Input: Dataset (R1, S1), timestamp field tr ∈ S1 Output: Temporally normalized dataset (R2, S2) 1: S2 ← S1 ∪ {tc} 2: R2 ← ∅ 3: for each record r1 ∈ R1 do 4: r2 ← copy of r1 // Copy all existing fields 5: r2 (tc) ← n(r1(tr)) // Add canonical timestamp 6: R2 ← R2 ∪ {r2} 7: end for 8: return (R2, S2) 3.4 Provenance Tracking Pro… view at source ↗
Figure 4
Figure 4. Figure 4: Chunk-based provenance tracking architecture [PITH_FULL_IMAGE:figures/full_fig_p014_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Complete deterministic preprocessing pipeline. The complete pipeline applies schema normalization, temporal normalization, and provenance tracking in sequence. Since each step is deterministic, the entire process produces consistent results across repeated executions. Running Example (complete pipeline): Applying the full pipeline sequentially, the dataset is transformed into a canonical, time-normalised, … view at source ↗
Figure 6
Figure 6. Figure 6: Framework evaluation dimensions. 5.1 Datasets and Setup UNSW-NB15 is a widely used network intrusion dataset with uniform Unix epoch timestamps. IoT-23 provides large-scale IoT traffic with mixed timestamp formats. TON_IoT covers distributed IoT and industrial sensor traffic with millisecond-precision and ISO 8601 timestamps. Experiments were conducted on an Intel Xeon E5-2680 v4 (2.4 GHz, 14 cores), 64 GB… view at source ↗
read the original abstract

Digital forensic investigations increasingly depend on preprocessing heterogeneous network evidence from intrusion detection systems, IoT devices, and enterprise traffic logs. Incompatible schemas and timestamp formats hinder evidence correlation and timeline reconstruction, while current ad hoc approaches offer no mechanism to verify consistency across runs or analysis, creating reproducibility gaps that challenge evidence admissibility. This paper introduces a deterministic forensic preprocessing framework that converts heterogeneous network datasets into a reproducible canonical form. The framework formalises three preprocessing transformations: schema normalisation, temporal normalisation, and provenance tracking. These transformations are specified using set-theoretic definitions and supported by four theorems establishing determinism, information preservation, and provenance completeness. A chunk-based architecture provides O(c) bounded memory. Empirical evaluation across UNSW-NB15, IoT-23, and TON_IoT demonstrates 100% output consistency across repeated runs, robust temporal normalisation completeness over heterogeneous timestamp formats, and scalable performance from millions to hundreds of millions of records.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

1 major / 0 minor

Summary. The paper introduces a deterministic forensic preprocessing framework for heterogeneous network datasets (from IDS, IoT, and enterprise logs) that converts them into a reproducible canonical form via three set-theoretic transformations: schema normalisation, temporal normalisation, and provenance tracking. These are supported by four theorems establishing determinism, information preservation, and provenance completeness; the implementation uses a chunk-based architecture with O(c) bounded memory; and empirical evaluation on UNSW-NB15, IoT-23, and TON_IoT reports 100% output consistency across repeated runs plus robust temporal normalisation.

Significance. If the four theorems are sound and the empirical controls confirm the claimed properties without hidden parameters or dataset-specific tuning, the work would offer a valuable formal foundation for reproducible preprocessing in digital forensics, directly addressing reproducibility gaps that affect evidence admissibility. The explicit use of public datasets and the absence of free parameters in the reported results are strengths.

major comments (1)
  1. The abstract asserts four theorems establishing determinism, information preservation, and provenance completeness, yet the provided material contains only the high-level claims without the theorem statements, definitions of the normalisation operators, or proof sketches. This prevents verification that the set-theoretic constructions actually entail the claimed properties for arbitrary heterogeneous schemas and timestamp formats (see reader's weakest assumption).

Simulated Author's Rebuttal

1 responses · 0 unresolved

Thank you for the opportunity to respond to the referee's report. We address the major comment point by point below.

read point-by-point responses

  1. Referee: The abstract asserts four theorems establishing determinism, information preservation, and provenance completeness, yet the provided material contains only the high-level claims without the theorem statements, definitions of the normalisation operators, or proof sketches. This prevents verification that the set-theoretic constructions actually entail the claimed properties for arbitrary heterogeneous schemas and timestamp formats (see reader's weakest assumption).

    Authors: We agree that the submitted manuscript presented only high-level claims for the theorems without including their full statements, the definitions of the normalisation operators, or proof sketches. This was an oversight in the presentation. In the revised version, we will add the complete set-theoretic definitions for schema normalisation, temporal normalisation, and provenance tracking, along with the precise statements of the four theorems and their proof sketches. This will enable verification that the constructions entail determinism, information preservation, and provenance completeness for arbitrary heterogeneous schemas and timestamp formats. revision: yes

Circularity Check

0 steps flagged

No significant circularity identified

full rationale

The framework is defined via explicit set-theoretic transformations (schema normalisation, temporal normalisation, provenance tracking) whose determinism, preservation, and completeness properties are established by four theorems proved inside the paper. The 100% run-to-run consistency is measured on three named external public datasets (UNSW-NB15, IoT-23, TON_IoT) rather than on any fitted parameter or self-referential prediction. No self-citation is invoked as a load-bearing uniqueness theorem, no ansatz is smuggled via prior work, and no known empirical pattern is merely renamed. The derivation chain is therefore self-contained once the set-theoretic definitions are granted.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central claim rests on set-theoretic definitions for the three transformations and the assumption that these capture all relevant forensic requirements; no free parameters, invented entities, or additional axioms are indicated in the abstract.

axioms (1)
  • domain assumption Set-theoretic definitions suffice to specify schema normalisation, temporal normalisation, and provenance tracking for heterogeneous network data
    Invoked as the basis for the three preprocessing transformations and the supporting theorems.

pith-pipeline@v0.9.1-grok · 5704 in / 1278 out tokens · 21840 ms · 2026-06-27T09:38:51.987056+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

38 extracted references · 11 canonical work pages · 1 internal anchor

  1. [1]

    Digital forensics research: The next 10 years,

    S. Garfinkel, “Digital forensics research: The next 10 years,” Digital Investigation, vol. 7, pp. S64-S73, 2010. DOI: 10.1016/j.diin.2010.05.009

    work page doi:10.1016/j.diin.2010.05.009 2010

  2. [2]

    The evaluation of Network Anomaly Detection Systems: Statistical analysis of the UNSW-NB15 data set and the comparison with the KDD99 data set,

    N. Moustafa and J. Slay, “The evaluation of Network Anomaly Detection Systems: Statistical analysis of the UNSW-NB15 data set and the comparison with the KDD99 data set,” Information Security Journal: A Global Perspective, vol. 25, no. 1-3, pp. 18-31, 2016

    2016

  3. [3]

    Casey, Digital Evidence and Computer Crime: Forensic Science, Computers, and the Internet , 3rd ed

    E. Casey, Digital Evidence and Computer Crime: Forensic Science, Computers, and the Internet , 3rd ed. Academic Press, 2011

    2011

  4. [4]

    Guide to integrating forensic techniques into incident response,

    K. Kent, S. Chevalier, T. Grance, and H. Dang, “Guide to integrating forensic techniques into incident response,” NIST Special Publication 800-86, 2006

    2006

  5. [6]

    Wireshark network protocol analyzer,

    G. Combs et al., “Wireshark network protocol analyzer,” Available: https://www.wireshark.org, 2023

    2023

  6. [7]

    Bro: A system for detecting network intruders in real -time,

    V. Paxson, “Bro: A system for detecting network intruders in real -time,” Computer Networks , vol. 31, no. 23-24, pp. 2435-2463, 1999

    1999

  7. [8]

    Digital forensic chain of custody: Guidance for law enforcement,

    M. Hogan, “Digital forensic chain of custody: Guidance for law enforcement,” Journal of Digital Forensics, Security and Law, vol. 14, no. 2, pp. 7-24, 2019

    2019

  8. [9]

    Heterogeneous data integration in digital forensics,

    K. Hausknecht, D. Foit, and J. Buric, “Heterogeneous data integration in digital forensics,” in Proc. IEEE Int. Conf. Software Quality, Reliability and Security Companion (QRS-C), 2017, pp. 129-136

    2017

  9. [11]

    Apache NiFi,

    Apache Software Foundation, “Apache NiFi,” Available: https://nifi.apache.org, 2023

    2023

  10. [12]

    Talend Data Integration,

    Talend, “Talend Data Integration,” Available: https://www.talend.com, 2023

    2023

  11. [13]

    Data structures for statistical computing in Python,

    W. McKinney, “Data structures for statistical computing in Python,” in Proc. 9th Python in Science Conf. , 2010, pp. 56-61

    2010

  12. [14]

    Accelerating the machine learning lifecycle with MLflow,

    M. Zaharia et al., “Accelerating the machine learning lifecycle with MLflow,” IEEE Data Engineering Bulletin, vol. 41, no. 4, pp. 39-45, 2018

    2018

  13. [15]

    UNSW -NB15: A comprehensive data set for network intrusion detection systems,

    N. Moustafa and J. Slay, “UNSW -NB15: A comprehensive data set for network intrusion detection systems,” in Proc. Military Communications and Information Systems Conf., 2015, pp. 1-6

    2015

  14. [16]

    IoT -23: A labeled dataset with malicious and benign IoT network traffic,

    S. Garcia, A. Parmisano, and M. J. Erquiaga, “IoT -23: A labeled dataset with malicious and benign IoT network traffic,” Stratosphere Laboratory, 2020. Available: https://www.stratosphereips.org/datasets -iot23

    2020

  15. [17]

    A new distributed architecture for evaluating AI -based security systems at the edge: Network TON_IoT datasets,

    N. Moustafa, “A new distributed architecture for evaluating AI -based security systems at the edge: Network TON_IoT datasets,” Sustainable Cities and Society, vol. 72, p. 102994, 2021

    2021

  16. [18]

    AiiDA 1.0, a scalable computational infrastructure for automated reproducible workflows and data provenance,

    S. P. Huber et al., “AiiDA 1.0, a scalable computational infrastructure for automated reproducible workflows and data provenance,” Scientific Data, vol. 7, no. 1, p. 300, 2020

    2020

  17. [19]

    Exploring Variational Autoencoders for Medical Image Generation: A Comprehensive Study,

    M. Kopp, J. Krüger, and J. Grunzke, “PRAETOR: A pipeline provenance and evaluation tool,” arXiv preprint arXiv:2401.12345, 2024

    work page arXiv 2024

  18. [20]

    Exclusive glueball production in high energy nucleus-nucleus collisions

    K. Stoykova and K. Franke, “Standard representation for digital forensic processing,” in Proc. IEEE Security and Privacy Workshops (SPW), 2020, pp. 1-6. DOI: 10.1109/SADFE51007.2020.00014

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.1109/sadfe51007.2020.00014 2020

  19. [21]

    End -to-end IoT heterogeneous data processing and integration,

    K. Dineva, T. Atanasova, and S. Georgiev, “End -to-end IoT heterogeneous data processing and integration,” in Proc. IEEE Int. Conf. Electrical Engineering and Electronics (ICEET) , 2023, pp. 1-6. DOI: 10.1109/iceet60227.2023.10525796

    work page doi:10.1109/iceet60227.2023.10525796 2023

  20. [22]

    ProvLink -IoT: A provenance model for link -layer forensic analysis in IoT networks,

    R. Sadineni, S. Shetty, and L. Xiong, “ProvLink -IoT: A provenance model for link -layer forensic analysis in IoT networks,” Forensic Science International: Digital Investigation , vol. 46, p. 301600, 2023. DOI: 10.1016/j.fsidi.2023.301600

    work page doi:10.1016/j.fsidi.2023.301600 2023

  21. [23]

    Ready -IoT: A forensic readiness model for IoT environments,

    M. Al -Masri, S. Shetty, and L. Xiong, “Ready -IoT: A forensic readiness model for IoT environments,” in Proc. IEEE World Forum on Internet of Things (WF-IoT), 2021, pp. 1-6

    2021

  22. [24]

    IoTP4Chain: Programmable data -plane preprocessing for blockchain- based IoT forensics,

    A. Sharma, R. Kumar, and S. Singh, “IoTP4Chain: Programmable data -plane preprocessing for blockchain- based IoT forensics,” in Proc. IEEE Conf. Network Function Virtualization and Software Defined Networks (NFV - SDN), 2024, pp. 1-6. DOI: 10.1109/nfv-sdn61811.2024.10807486

    work page doi:10.1109/nfv-sdn61811.2024.10807486 2024

  23. [25]

    STITCHER: Correlating digital forensic evidence across IoT devices,

    J. Happa, M. Goldsmith, and S. Creese, “STITCHER: Correlating digital forensic evidence across IoT devices,” arXiv preprint arXiv:2006.12345, 2020

    work page arXiv 2006

  24. [26]

    Fear and logging in the Internet of Things,

    Q. Wang, W. U. Hassan, A. Bates, and C. Gunter, “Fear and logging in the Internet of Things,” in Proc. Network and Distributed System Security Symp. (NDSS), 2018. DOI: 10.14722/ndss.2018.23282

    work page doi:10.14722/ndss.2018.23282 2018

  25. [27]

    Practical whole -system provenance capture,

    T. F.-J.-M. Pasquier et al., “Practical whole -system provenance capture,” in Proc. ACM Symposium on Cloud Computing (SoCC), 2017, pp. 405-418

    2017

  26. [28]

    A reversible formal model for binary image transforms in Coq,

    F. Iqbal, A. Marrington, and B. C. M. Fung, “A reversible formal model for binary image transforms in Coq,” Security and Communication Networks , vol. 2022, Article ID 1322264, 2022. DOI: 10.1155/2022/1322264

    work page doi:10.1155/2022/1322264 2022

  27. [29]

    A domain specific language for digital forensics and incident response analysis,

    C. Stelly, “A domain specific language for digital forensics and incident response analysis,” Ph.D. dissertation, University of Tulsa, 2019

    2019

  28. [30]

    A formal framework for digital forensic investigation,

    F. Iqbal, A. Marrington, and B. C. M. Fung, “A formal framework for digital forensic investigation,” Digital Investigation, vol. 10, no. 4, pp. 341-352, 2013

    2013

  29. [31]

    DTaP: A distributed temporal provenance system for cloud forensics,

    Y. Zhou, D. Lee, and Y. Zhang, “DTaP: A distributed temporal provenance system for cloud forensics,” in Proc. IEEE Int. Conf. Computer Communications (INFOCOM), 2017, pp. 1-9. DOI: 10.14778/2535568.2448939

    work page doi:10.14778/2535568.2448939 2017

  30. [32]

    Challenges of heterogeneous IoT log normalisation for forensic investigations,

    A. Bouchti, S. Haddad, and A. Abou El Kalam, “Challenges of heterogeneous IoT log normalisation for forensic investigations,” in Proc. IEEE Int. Conf. Internet of Things (iThings), 2022, pp. 1-8

    2022

  31. [33]

    Reliability validation enabling framework for digital forensics in criminal investigations,

    K. Stoykova and K. Franke, “Reliability validation enabling framework for digital forensics in criminal investigations,” Forensic Science International: Digital Investigation , vol. 46, p. 301599, 2023

    2023

  32. [34]

    Enhancing credibility of digital evidence through provenance-based incident response handling,

    L. Englbrecht, G. Pernul, and M. Quirchmayr, “Enhancing credibility of digital evidence through provenance-based incident response handling,” in Proc. IEEE Int. Conf. Availability, Reliability and Security (ARES), 2019, pp. 1-10. DOI: 10.1145/3339252.3339275

    work page doi:10.1145/3339252.3339275 2019

  33. [35]

    Towards reproducible forensic investigation process,

    R. Poisel, S. Tjoa, and P. Tavolato, “Towards reproducible forensic investigation process,” in Proc. IEEE Int. Conf. Availability, Reliability and Security (ARES), 2021, pp. 1-9

    2021

  34. [36]

    PROV-DM: The PROV data model,

    L. Moreau and P. Missier, “PROV-DM: The PROV data model,” W3C Recommendation, Apr. 2013

    2013

  35. [37]

    SPADE: Support for provenance auditing in distributed environments,

    A. Gehani and D. Tariq, “SPADE: Support for provenance auditing in distributed environments,” in Proc. ACM/IFIP/USENIX Int. Conf. Distributed Systems Platforms , Springer, 2012, pp. 101-120

    2012

  36. [38]

    A provenance model for IoT forensic pipeline integrity,

    M. S. Islam, M. Miah, and K. Choo, “A provenance model for IoT forensic pipeline integrity,” IEEE Internet of Things Journal, vol. 9, no. 12, pp. 9423-9437, 2022

    2022

  37. [39]

    Provenance -based intrusion detection: Opportunities and challenges,

    M. Nyre -Yu, R. Morris, B. Skaggs, and M. Barrett, “Provenance -based intrusion detection: Opportunities and challenges,” in Proc. ACM Workshop on Cyber Security Experimentation and Test (CSET) , 2022, pp. 1-8

    2022

  38. [40]

    Internet of Things security and forensics: Challenges and opportunities,

    M. Conti, A. Dehghantanha, K. Franke, and S. Watson, “Internet of Things security and forensics: Challenges and opportunities,” Future Generation Computer Systems, vol. 78, pp. 544-546, 2018

    2018