PHASE: Passive Human Activity Simulation Evaluation

Anh Nguyen-Tuong; Jack W. Davidson; Jason D. Hiser; Steven Lamp

PHASE: Passive Human Activity Simulation Evaluation

T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read PHASE is a passive framework that claims to distinguish human from non-human network activity with over 90% accuracy and to turn the result into a fidelity score for synthetic user personas.

desk verdict PHASE's headline claim is undercut by a device-class confound in its labels; the datasets and framework are real, but the MCHP case study needs a control. read the letter →

arxiv 2507.13505 v2 pith:H3CMIJ2Q submitted 2025-07-17 cs.CR cs.AIcs.LGcs.NI

Steven Lamp , Jason D. Hiser , Anh Nguyen-Tuong , Jack W. Davidson This is my paper · ORCID

classification cs.CRcs.AIcs.LGcs.NI

keywords passivenetworkmonitoringbehavioralfidelitysyntheticuserpersonasZeekconnectionlogsdeepneuralclassificationSHAPexplainabilityhumanactivityrecognitioncyberranges

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

PHASE addresses a gap in cybersecurity simulation: there is no standardized quantitative measure of how closely synthetic user personas resemble real human network behavior. The paper claims that a deep neural network trained on Zeek connection logs can distinguish human-generated from automated network activity with over 90% accuracy, using only passive monitoring, and that the model's output can serve as a behavioral fidelity score. This matters because cyber ranges, honeypots, and sandboxes rely on realistic user traffic to train and test defenses, and currently have no principled way to tune or compare synthetic personas. The paper demonstrates the score's diagnostic value by showing that the default Caldera Human Plugin reads as non-human, then improves the persona by adding one-hour idle periods and measuring a higher human-likeness score.

What carries the argument

The load-bearing mechanism is the PHASE scoring pipeline. Raw Zeek connection logs are aggregated per device into daily sequences of 1,440 one-minute bins; within each bin, numeric features are averaged, categorical features are integer-encoded, missing values are filled with -1, and all numerical features are min-max scaled. The classifier is a hybrid deep network: a 1D convolutional layer feeding two bidirectional long short-term memory layers, each followed by a multi-headed attention layer and dropout, trained with stratified 10-fold cross-validation and undersampling of the majority class. The model's output probability for a day is the PHASE score. Training labels are assigned by an expert panel that reads DNS tables and IP assignment records: desktops and laptops count as human; servers, GPU virtual machines, printers, and IoT devices count as non-human.

What would settle it

Run the same synthetic persona with identical workflows on an ordinary desktop or laptop instead of a server virtual machine and compare PHASE scores; if the score rises into the human range, the classifier is detecting hardware class rather than human behavioral realism.

Watch

Extended reading notes

Core claim

The central claim is that a model trained only on passively observed Zeek connection metadata can separate human from non-human network activity with over 90% accuracy, and that the resulting probability score is a usable fidelity metric for synthetic user personas. The paper supports this with three datasets collected from a live academic network across three terms, labeled by an expert panel using DNS tables and IP assignment records. In a case study, the default Caldera Human Plugin scores 0.24 to 0.31 across models, far below the human threshold, while a modified configuration with randomized one-hour idle periods scores noticeably higher. SHAP analysis attributes the gap to low connection volume and packet counts, plus feature patterns at specific times of day.

Load-bearing premise

The load-bearing premise is that the expert-panel device labels really separate human from non-human activity: desktops and laptops are treated as human, while servers, printers, IoT devices, and virtual machines are treated as non-human, so the model may be learning device class rather than a general concept of human behavior.

Editorial extensions

If this is right

Any cyber range or honeypot that can capture Zeek logs can produce a daily human-likeness score for its synthetic personas without adding user-side instrumentation.
The default Caldera Human Plugin configuration scores 0.24 to 0.31 across the three models, placing it below the human threshold and quantifying the long-suspected gap in synthetic user realism.
Inserting randomized one-hour idle periods into the persona raises PHASE scores across all three datasets, giving synthetic persona designers a concrete, measured adjustment strategy.
SHAP analysis points to connection state, missed bytes, and history strings as the most influential features, so improvements should target connection-level exchange patterns rather than traffic volume alone.
The three labeled CSNET datasets, spanning summer, fall, and spring terms with 187 human IP addresses, provide a reusable benchmark for future fidelity-measurement research.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the device-class entanglement is real, the reported accuracy overstates behavioral detection; a stricter evaluation would hold the hardware fixed and vary only whether activity is scripted or human-driven.
The same passive labeling recipe could be inverted to score whether an attacker's interaction with a honeypot appears human-operated, a quantity directly relevant to engagement quality.
Treating the PHASE score as an objective function would let synthetic persona designers search over behavioral parameters such as idle timing, burstiness, and application mix, rather than hand-tuning from SHAP plots alone.
Extending the framework to finer non-human classes, which the paper lists as future work, would turn the same telemetry into a general taxonomy of automation instead of a single human/non-human split.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Two things worth knowing before you read arXiv:2507.13505. First, the authors have built a real asset: three labeled Zeek-log datasets from a campus network, covering roughly 291 days and 187 human-labeled devices, with a DNS-based labeling process that is practical and clearly described. Second, the headline claim—>90% accuracy distinguishing human from non-human activity—does not hold up under scrutiny, because the labels perfectly encode device class. Human activity is restricted to desktops and laptops; non-human includes servers, VMs, printers, and IoT. The classifier can reach 95% by learning 'client vs server' rather than any general human concept. The case study then runs MCHP on 14 Ubuntu Server VMs, which fall in the non-human class, so the low PHASE scores say more about the deployment hardware than about behavioral realism. The SHAP features that dominate (norm_vol, resp_pkts, orig_ip_bytes) are textbook server-vs-client discriminators.

What the paper does well: the DNS-to-device labeling is a genuine contribution, the datasets fill a real gap, the survey of why prior datasets don't fit is useful, and the writing is clear. The SHAP-guided tuning workflow is a nice idea even if its output is confounded.

The soft spots beyond the central one: the cross-validation split is not described at device level. They say IPs were used for labeling and splitting, but if daily sequences from the same IP appear in both train and test folds, the reported accuracies could be inflated. The tuning loop is also circular: the 'enhanced' configuration comes from SHAP on the same model, then is scored by that same model. Higher scores are unsurprising; they show model-likeness, not human-likeness. No independent model or human judgment backs it up. Also, no code or data are released, which is a problem since the datasets are the main asset. Minor: the threshold bullet '0.6 ≤ S ≥ 0.4' is mathematically garbled, and some numbers are rounded oddly, but those are cosmetic.

Who this is for: people building or evaluating synthetic user personas in cyber ranges. The framework is plausible and the datasets are valuable. The paper just doesn't yet support its main claim. The fix is specific: control for device class (e.g., run MCHP on desktop-class endpoints), and group cross-validation folds by device. Those are addressable in revision.

Recommendation: yes, send to peer review. The idea is useful, the data is a real contribution, and the flaws are fixable. A serious referee could turn this into a solid paper.

Referee Report

3 major / 4 minor

Summary. PHASE is a passive machine-learning framework that classifies Zeek connection logs as human or non-human network activity, with the stated purpose of scoring the behavioral fidelity of synthetic user personas. The authors collected three large academic-network datasets, labeled devices as human or non-human through an expert-panel review of DNS and IP records, and trained hybrid CNN/BiLSTM/attention models that reportedly achieve over 90% accuracy and balanced accuracy. They apply SHAP to interpret the models, then use the resulting feature insights to modify the MITRE Caldera Human Plugin default configuration, reporting that the enhanced configuration receives higher PHASE scores. The paper claims this is the first quantitative, passive fidelity metric for synthetic user personas.

Significance. If the central claim holds, PHASE addresses a real gap: cyber ranges, honeypots, and sandboxes lack a standardized quantitative way to evaluate whether synthetic user behavior resembles real human activity. The paper's strengths include the use of three large, real-world network datasets collected over entire academic terms, the passive measurement design using a Zeek appliance, explicit IRB review and IP anonymization, and the use of SHAP to expose which traffic features drive classification. The framework's intended application to iterative SUP tuning is also practically valuable. However, the reported accuracy and the case-study conclusions rest on label definitions that confound device class with human activity, and on a data-splitting procedure whose leakage controls are not fully specified; these issues must be addressed before the central claim is supported.

major comments (3)

[§3.2.1–3.2.2, §4.1] The label construction perfectly confounds device class with the target concept. Section 3.2.1 restricts human-labeled devices to desktops and laptops, while Section 3.2.2 places servers, GPU VMs, printers, IoT devices, and single-board computers into the non-human class. Because the two classes are separated by device type, a model can achieve high accuracy by learning a server-vs-client discriminator rather than a general human-activity detector. The case study compounds this problem: Section 4.1 deploys MCHP on 14 Ubuntu Server 23.10 virtual machines, exactly the device class labeled non-human. The low PHASE scores in Section 4.2.1 may therefore reflect that MCHP runs on server-class VMs, not that its behavior is behaviorally non-human. The SHAP findings in Section 3.6.1, which emphasize volume and packet features that differ systematically between servers and end-user machines, are consistent with this alternative explanation. I request a concrete control: evaluate the MCHP persona on desktop/laptop endpoints and compare with its server-VM scores, and additionally report classifier performance when training and test sets are stratified by device type (or show that the model generalizes across device classes). Without such an analysis, the headline accuracy claim and the MCHP evaluation do not establish behavioral-realism measurement.
[§2.2.1, §2.3, Table 4] The data-partitioning scheme is underspecified in a way that is load-bearing for the reported accuracies. Section 2.2.1 states that each device's activity is divided into daily sequences, and Section 2.3 describes stratified 10-fold cross-validation, but the manuscript does not say whether folds are split at the level of IP/device or at the level of individual daily sequences. If daily sequences from the same device appear in both training and validation folds, temporally autocorrelated behavior and device-specific patterns will leak across folds, artificially inflating the accuracy, balanced accuracy, and AUC values in Table 4. The authors should specify the split granularity and, if it is not already device-grouped, repeat the evaluation with group-wise cross-validation where all days from a given IP are confined to a single fold.
[§4.2.2, §3.5] The 'enhanced' MCHP configuration is derived from SHAP analysis of the same PHASE model that is then used to score it, creating a closed loop that can reward fitting the model's learned proxies rather than improving genuine human realism. Section 4.2.2 says the revised configuration was 'guided by insights from SHAP-based feature analysis,' and Figure 10 reports higher PHASE scores for the enhanced configuration on the same models used to produce those SHAP values. Because SHAP explains a specific model's decision boundary, tuning to that explanation and then measuring with the same model is not an independent validation of increased behavioral fidelity. I recommend evaluating the enhanced SUP on held-out PHASE models (e.g., models trained on the other two datasets, or a fresh data-collection period) and, if possible, comparing against a small human-ground-truth dataset to confirm that the changed behavior corresponds to real human activity rather than to model-specific artifacts.

minor comments (4)

[§4.2] The threshold definitions contain an inconsistent inequality: the ambiguous band is written as '0.6 ≤ S ≥ 0.4,' which is not a valid interval; it should be '0.4 ≤ S < 0.6.'
[§2.2.1] The statement that minute-level intervals were selected 'after testing coarser options, such as hourly aggregation' is not accompanied by any quantitative comparison; reporting the accuracy or loss under hourly aggregation would make the modeling choice reproducible.
[§2.2.2] Using LabelEncoder for categorical features such as connection state and history imposes an arbitrary ordinal relationship among categories; a one-hot or embedding representation would avoid this assumption, and the choice should at least be discussed.
[§3.1.2, Table 3] The datasets are described as 'over 100 GB each,' but Table 3 reports only connection-log counts; stating the exact PCAP-to-log reduction or the log sizes would help readers assess the volume of data used for training.

Circularity Check

2 steps flagged · score 6.0 of 10

MCHP case study is non-human by label definition, and the 'enhanced' SUP is tuned and scored by the same PHASE model, making the central demonstration partially self-referential.

self definitional [Sections 3.2.2 and 4.1 (label definitions and MCHP deployment)]
"Devices classified as non-human included printers, GPU-based virtual machines used for experimental deployments, server clusters, IoT devices (e.g., thermostats)... For this case study, MCHP was deployed on 14 Ubuntu Server 23.10 virtual machines within an isolated internal server cluster."

Human activity is defined by device type: Section 3.2.1 restricts human labels to 'desktop computers and laptops', while Section 3.2.2 places 'server clusters' and 'GPU-based virtual machines' in the non-human class. The case-study SUP runs on 14 Ubuntu Server VMs, a server-cluster deployment that falls in the non-human class by definition. Therefore the low PHASE scores for default MCHP are entailed by the choice of server VMs, not by an independent measurement of behavioral realism. The classifier can score >90% accuracy by separating server-like from client-like traffic, so the headline claim and the MCHP evaluation do not establish a device-independent 'human activity' concept.
fitted input called prediction [Section 4.2.2]
"To improve the behavioral realism of MCHP, we introduced a modified configuration, referred to as MCHP Enhanced, guided by insights from SHAP-based feature analysis. ... As shown in Figure 10, the enhanced configuration yields consistently higher PHASE scores (blue bars) across all models, indicating improved alignment with human-like activity."

The SHAP analysis used to select the enhancement is computed from the PHASE v29.4X model, and the resulting MCHP Enhanced configuration is then scored by the same PHASE models. The improvement is therefore a measure of how well the configuration satisfies the evaluator's own feature preferences, not an independent test of human-likeness. No held-out human ground truth or external fidelity metric is used; the tuning signal and the evaluation metric are the same model. This makes the 'significantly improves the human-likeness' claim partially self-referential, though the classification accuracy itself is trained on real labels.

full rationale

The core classification accuracy is not circular: the PHASE models are trained on real-world connection logs with labels assigned by an expert panel, so the 90%+ accuracy reflects genuine separation of the two label classes in that data. However, the paper's central demonstration—the MCHP case study—contains two reducing steps. First, the label definitions align almost perfectly with device class: human means desktops/laptops, non-human includes servers and GPU VMs, and MCHP is deployed on Ubuntu Server VMs. Thus the low default-MCHP scores are predetermined by the deployment platform, not by an independent measure of behavioral realism. Second, the 'enhanced' MCHP configuration is chosen using SHAP explanations from the same PHASE model that then reports the improved scores, so the claimed improvement is an optimization against the evaluator's own preferences rather than validation against external human ground truth. These issues do not make the training accuracy circular, but they do make the paper's headline claim and its SUP-tuning demonstration partially self-referential. No load-bearing self-citations or imported uniqueness theorems appear; the SENTINEL citation is background architecture, not an argument. The score reflects the two construction-level reductions affecting the paper's main evaluation claims.

Assumptions & free parameters 7 free parameters · 4 assumptions · 0 invented entities

The framework rests on a set of domain-specific modeling choices and expert-labeling axioms. No new physical or conceptual entities are invented. The main burdens are the device-class proxy for human activity, the unspecified split details, and the many manual hyperparameter choices.

free parameters (7)

Minute-level aggregation interval = 1 minute
Connection logs are aggregated per device into 1,440 one-minute time bins per day; the choice was made after testing hourly aggregation and affects the input sequence length and model behavior.
Training epochs = 1,000
Model is trained for up to 1,000 epochs with early stopping and restores best weights; the epoch limit is a manual choice.
Learning rate = 0.001
Used with Adamax optimizer, selected 'to balance training stability with computational efficiency' (Section 2.3).
Dropout rate = 0.2
Dropout layers after LSTM and attention layers use rate 0.2 to prevent overfitting.
Missing value fill = -1
Missing minute-level features are filled with -1, a value chosen to fall outside the label encoding range and survive normalization.
Fidelity score thresholds = 0.5 and 0.8/0.6/0.4/0.2 bands
The mapping of PHASE score S to human-likeness categories is defined using the lowest observed model accuracy (93.9%), making the thresholds dependent on the model's own performance.
SHAP analysis settings = not specified
The computation of SHAP values (background sample size, approximation method) is not reported, yet the case-study conclusions depend on these visualizations.

assumptions (4)

domain assumption The five-expert panel correctly identifies human-operated devices versus automated devices from IP assignment and DNS records.
Section 3.2.1 and 3.2.2 rely entirely on expert judgment to assign device labels; if these labels are noisy, the ground truth for the classifier is unreliable.
ad hoc to paper Device class (laptop/desktop vs. server/IoT) is a valid proxy for human vs. non-human activity.
The labels equate 'human' with workstations and 'non-human' with servers, VMs, printers, and IoT devices. This conflates usage behavior with device type and is particularly problematic in the MCHP case study, where the SUP runs on VMs.
domain assumption The data partitioning keeps all daily sequences for a given device in the same fold, preventing leakage.
Section 2.2.1 says IP addresses are used for splitting, but the exact fold assignment per device is not specified. If daily sequences from one device appear in both train and validation, accuracy would be inflated.
domain assumption Zeek connection log metadata contains sufficient signal to infer human interaction.
The entire feature set consists of connection-level aggregates (counts, durations, ports, state strings). If human behavior is not visible in these metadata, the approach cannot work.

how reviews work

0 comments

Cite this review

Pith. "Pith review of PHASE: Passive Human Activity Simulation Evaluation." pith.science (2026). https://pith.science/paper/H3CMIJ2Q

@misc{pith2026250713505,
  author       = {Pith},
  title        = {Pith review of: PHASE: Passive Human Activity Simulation Evaluation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/H3CMIJ2Q}},
  note         = {Machine review of arXiv:2507.13505}
}

read the original abstract

Cybersecurity simulation environments, such as cyber ranges, honeypots, and sandboxes, require realistic human behavior to be effective, yet no quantitative method exists to assess the behavioral fidelity of synthetic user personas. This paper presents PHASE (Passive Human Activity Simulation Evaluation), a machine learning framework that analyzes Zeek connection logs and distinguishes human from non-human activity with over 90\% accuracy. PHASE operates entirely passively, relying on standard network monitoring without any user-side instrumentation or visible signs of surveillance. All network activity used for machine learning is collected via a Zeek network appliance to avoid introducing unnecessary network traffic or artifacts that could disrupt the fidelity of the simulation environment. The paper also proposes a novel labeling approach that utilizes local DNS records to classify network traffic, thereby enabling machine learning analysis. Furthermore, we apply SHAP (SHapley Additive exPlanations) analysis to uncover temporal and behavioral signatures indicative of genuine human users. In a case study, we evaluate a synthetic user persona and identify distinct non-human patterns that undermine behavioral realism. Based on these insights, we develop a revised behavioral configuration that significantly improves the human-likeness of synthetic activity yielding a more realistic and effective synthetic user persona.