{"id":"c8a63d6f-7559-42dd-ac7e-6f6dfb989f67","arxiv_id":"2502.05367","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"EarlyCrow detects APT malware command-and-control over HTTP(S) by classifying contextual summaries of network flows, achieving a macro F1 of about 93% on unseen APT families.","lead":"This paper proposes EarlyCrow, a machine-learning detector that spots command-and-control traffic from advanced persistent threat (APT) malware inside HTTP(S) connections. It works by building a rich 'contextual summary' of a network capture, combining flow, host, destination, and URL-level features, and reports high accuracy on malware families it was never trained on.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline 93% mF1 rests on an unverified platform-invariance assumption: APT and legitimate samples come from the same sandbox/CTU sources, and Section VI-B's claim that acquisition shift is mitigated by feature selection is never quantitatively tested.","rationale":"The reader's weakest assumption is that sandbox captures represent real enterprise APT C&C traffic well enough for the held-out F1 to generalize; my review identifies the same load-bearing point and sharpens it. The paper explicitly names acquisition shift in Section VI-B1 and claims to control it in Section VI-B2, but the control is implemented as a feature-selection policy, not validated by a quantitative invariance check. Because both positive and negative samples are produced in the same sandbox/CTU environment, any residual platform artifact can serve as a shortcut feature that raises held-out performance without helping in real deployment. The proposed test, a platform classifier plus a strict train-on-one-source/test-on-the-other evaluation, would settle whether the shortcut exists. If the cross-platform mF1 is close to the reported numbers, the generalization claim is credible; if it drops, the appropriate conclusion is that EarlyCrow is promising but the headline needs qualification. This matches the reader's conditional verdict exactly, so no verdict adjustment is needed. I also note the paper's own Limitations section (VII-H) recommends deployment evaluation, which supports treating the current numbers as sandbox-corpus evidence rather than proven field performance.","tokens_in":33627,"tokens_out":8109,"duration_ms":95251,"concrete_test":"Use the released features and labels to run two linked checks. First, train a binary classifier to predict the data source (Any.Run vs MCFP/CTU) from the Contextual Summary feature vector; if cross-validated AUC is materially above 0.5, the features are not invariant to platform, so acquisition shift persists. Second, re-run the unseen-APT evaluation with a strict platform split: train on Any.Run APT/legitimate samples only and test on MCFP/CTU samples only, and the reverse. Report macro F1 and FPR for both directions. If either direction drops substantially below 93%, the headline should be revised to describe sandbox-corpus performance, with deployment generalization treated as an open question.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that EarlyCrow generalizes to unseen APT malware. The evaluation uses PCAPs from Any.Run executions and MCFP/CTU captures (Section VI-A), and legitimate controls are drawn from the same sandbox environments. Section VI-B1 identifies acquisition shift as a bias, and Section VI-B2 says it is mitigated by keeping only 'domain-invariant representations' (e.g., dropping OS/browser/version names and TTL). That mitigation is asserted, not demonstrated: no invariance test, no feature-distribution comparison across platforms, and no cross-platform experiment is reported. Because training and test sets are both drawn from the same two sources, a model can achieve high held-out mF1 by learning Any.Run/CTU-specific artifacts, such as the 15-minute sandbox timing cutoff, single-host destination profiles, DNS request patterns, and UA popularity, that correlate with the label but may not transfer to an enterprise network. The 93.02% and 93.72% mF1 numbers are therefore conditional on the acquisition distribution; without evidence that the feature vector is truly independent of platform, the results mainly describe detection within the collected sandbox corpus rather than APT C&C detection in the wild. This is an external-validity concern, not an internal inconsistency, and the public code and causal analysis are real positives.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"EarlyCrow is a network-based detector for APT malware command-and-control (C&C) over HTTP(S). The paper first presents a threat model covering two HTTP(S)-based C&C cases and a set of MITRE ATT&CK TTPs, then introduces PairFlow, a context-preserving flow format that summarizes PCAP traffic into host, destination, URL, and HTTP(S) profiles. A random-forest classifier is trained on a ContextualSummary built from these profiles, using features motivated by a measurement study on APT, botnet, and legitimate traffic. The evaluation uses Any.Run sandbox captures and MCFP/CTU public captures, with experiments on known malware (10 random splits) and on unseen malware families held out from training, in both decrypted HTTP and opaque HTTPS modes. The headline results are a macro average F1 of 93.02% with FPR 0.74% for unseen APTs, and 93.72% mF1 in HTTPS mode.","tokens_in":33899,"tokens_out":5867,"duration_ms":62211,"significance":"If the results hold beyond the specific capture environments, this is a practically relevant contribution: it is one of few APT C&C evaluations on real malware traffic rather than simulated logs, and it comes with public code, datasets, and a clear family-split evaluation. The paper also gives a credible measurement study and a causal-analysis section that shows awareness of dataset bias, which is a strength for this area. The main limitation is external validity: training and test data are drawn from the same two platforms (Any.Run and MCFP/CTU), and the claim that acquisition shift is mitigated is asserted rather than demonstrated. The headline F1 numbers should therefore be read as conditional on the acquisition distribution until invariance is shown.","major_comments":[{"comment":"The claim that acquisition shift is mitigated is not backed by an empirical test. Section VI-B1 identifies platform configuration as a confounder and Section VI-B2 states that 'we consider only features with domain-invariant representations to ensure F ⊥⊥ Pf', listing a few dropped features (OS/browser/renderer names, TTL). Since both the training set and the unseen-family test set are drawn from the same Any.Run and MCFP/CTU platforms, a classifier could achieve the reported 93.02% mF1 by exploiting platform artifacts, such as the 15-minute sandbox truncation described in Section III, the single-host destination structure of sandboxed executions, or UA and DNS patterns specific to these capture sources. Please provide a quantitative invariance check, for example: (a) compare feature distributions stratified by platform; (b) train on Any.Run and test on MCFP/CTU and vice versa; or (c) show that the held-out family results are stable when each platform is removed in turn. Without such evidence, the headline 'unseen APT' claim remains an external-validity concern rather than a demonstrated generalization to enterprise traffic.","section":"Section VI-B1 and VI-B2"},{"comment":"The reported headline numbers are point estimates without measures of uncertainty. Table V is stated to be the average over ten random splits, but no standard deviation, confidence interval, or per-split range is given; Table VI appears to be a single split. The FPR of 0.74% and the mF1 differences between EarlyCrow and the baseline are therefore not statistically assessable. This matters for the central claim because the unseen-malware evaluation uses a small number of families and, in Table VII, per-malware detection rates are based on very few C&C servers (e.g., Zeus 3 servers, StrongPity 1 server). Please report mean ± std across multiple splits for Table VI, or provide exact binomial confidence intervals for the per-server detection rates, and state the number of test samples per class.","section":"Section VII-A, VII-B, Tables V and VI"},{"comment":"The HTTPS-mode experiment is underspecified. The text says that in HTTP mode the NIDS is connected to a web proxy that decrypts HTTPS, while in HTTPS mode it is placed at the network edge 'without deciphering HTTPS', but the paper never defines which features are available in HTTPS mode. Many ContextualSummary features in Table II depend on plaintext HTTP, such as UA strings (features 59-63), HTTP status-code ratios (10-13), content-type ratios (20-26), and URL-based features (82-102). Without an explicit feature-availability mask, the reader cannot tell whether the reported HTTPS mF1 of 93.72% comes from TLS metadata and traffic statistics or from plaintext features that would be unavailable in a real deployment. Please specify exactly which features are dropped or set to missing in HTTPS mode, and if possible validate the mask by comparing against a model that sees only TLS-level fields.","section":"Section VII, 'HTTPS-Mode' description"},{"comment":"The exclusion of 'samples without data transfer or failed TCP connections to C&C servers' is not quantified and may bias the malicious class toward easier, data-rich samples. The threat model in Section II explicitly includes fallback channels and low-volume C&C communications, so removing flows with no data transfer could preferentially remove exactly the stealthy cases the system is designed to detect. Please report how many samples and which families were removed by this filter, and provide a sensitivity analysis that includes or re-labels these samples instead of excluding them.","section":"Section VI-B2, data removal"}],"minor_comments":[{"comment":"The heading 'Architecutre Overview' contains a typo; it should read 'Architecture Overview'.","section":"Section IV heading"},{"comment":"The legend in Figure 7 uses inconsistent capitalization ('EarlyCrow' versus 'EARLYCROW' in the text and other figures); please unify.","section":"Figure 7"},{"comment":"The baseline is described only as 'inspired by MADE' with no precise feature list or hyperparameter settings. Since the baseline comparison is used to support the main claim, the configuration should be documented in enough detail to be reproducible.","section":"Section VII-B"},{"comment":"Several statements in the measurement study (e.g., '95.2% of APTs do not exceed a 0.19 DNS ratio') report percentages without sample sizes or confidence intervals; adding these would make the measurement study more interpretable.","section":"Section III and Figure 2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is an extended version of an ISC 2022 paper, and the authors clearly list the new material in footnote 2. My recommendation is based on the external-validity gap and the underspecified HTTPS mode; both are fixable with additional experiments or careful presentation. The public code and dataset are valuable assets, and I do not see an internal inconsistency that would warrant rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the core classifier and headline numbers are already out in the ISC 2022 paper; what this version adds is the PairFlow format, a 102-feature contextual-summary space, a measurement study of APT TTPs on a training set, family-separated evaluation, and a causal-analysis section that is honestly trying to address acquisition shift. That is a real engineering contribution, and the public code and data mean the work is reproducible, which counts for a lot.\n\nThe evaluation is genuinely careful in several places: train/test family separation, stratified splits, two operating modes (HTTP and HTTPS), and a per-family table that shows StrongPity at 0% detection over HTTPS. The authors do not hide the per-family failures, and that honesty is to their credit. The causal analysis in Section VI-B is a serious attempt, not a box-ticking paragraph.\n\nThe soft spots, in order of size:\n\n1. The platform-invariance claim is asserted, not demonstrated. Section VI-B2 says acquisition shift is mitigated by keeping \"domain-invariant representations\" and dropping TTL and OS/browser/version features, but there is no invariance test, no feature-distribution comparison across Any.Run and MCFP, and no cross-platform experiment. Because training and test both come from the same two sandbox/CTU sources, the model could be latching onto sandbox-specific artifacts (15-minute window, capture-timing patterns, UA popularity quirks) that correlate with the label but may not transfer. This is an external-validity concern, not an internal inconsistency, but it directly conditions the headline claim.\n\n2. Missing error bars. The known-malware 10-split averages in Table V have no variance or confidence intervals. The unseen-malware evaluation is a single split. These are cheap to fix and would substantially raise confidence.\n\n3. Selective exclusions, e.g., removing flows without data transfer or failed TCP connections, are reasonable but they are choices made after the fact, and the paper does not quantify how much of the raw corpus they remove.\n\nOn the plus side, the PairFlow format is a thoughtful answer to a real problem (NetFlow and Zeek logs drop the packet-level context that APT TTPs live in), and the feature-space diversity analysis (Section VII-C) and the TTP-feature correlation heatmaps are useful for anyone building on this line of work.\n\nBottom line: this is a credible, well-engineered paper with addressable weaknesses. It deserves a serious referee, and the authors should be pushed to temper the headline generalization claim, add error bars, and ideally run a small cross-platform or real-network validation. I would not cite it as proof that 93% mF1 holds in the wild, but I would cite it for PairFlow and the measurement study. I would bring it to reading group as a representative example of how to (and how not to) argue external validity in ML-based NIDS work.","headline":"Useful extended version of a solid ISC 2022 paper; the real weakness is that the headline 93% mF1 on unseen APTs rests on an asserted-but-unverified platform-invariance assumption, so the numbers likely describe the sandbox corpus better than real enterprise networks.","tokens_in":34463,"tokens_out":1249,"would_cite":true,"duration_ms":14162,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Contextual summaries let a detector catch APT command-and-control traffic it has never seen","keywords":["APT detection","command and control","network intrusion detection","HTTP(S) traffic","PairFlow","contextual summaries","random forest","threat model"],"falsifier":"Train EarlyCrow exactly as described, run it on packet captures from a real enterprise network with independently confirmed APT infections and manually verified benign traffic, and compare the macro F1 on families not in training; a material drop below the reported 93%, or a showing that platform-specific fields such as TTL dominate the decisions, would falsify the generalization claim.","tokens_in":33388,"feed_emoji":"🛡️","tokens_out":6198,"duration_ms":57929,"temperature":0.7,"pith_summary":"This paper claims that a network intrusion detector can recognize command-and-control (C&C) traffic from Advanced Persistent Threats (APTs) even when the malware family has never been seen during training. The key move is to stop classifying individual flows in isolation and instead build a contextual summary of everything a host does with a remote server over time, across DNS, HTTP(S), and raw TCP behavior. On held-out APT families, the resulting random-forest classifier reports a macro average F1 of 93.02% with a 0.74% false positive rate, and it keeps macro F1 near 93.72% when only opaque HTTPS traffic is visible. If this holds in deployment, organizations could catch APT C&C early without decrypting HTTPS.","feed_headline":"APT command-and-control detected in unseen malware at 93% F1","feed_subtitle":"A random forest over contextual network summaries catches APT families never seen in training, at 0.74% false positives.","key_machinery":"The load-bearing object is PairFlow, a flow format that groups every packet exchanged between one local host and one remote server within a time window, attaches the DNS requests and responses that resolved that server, and records HTTP fields, TLS cipher suites, packet planes, and per-packet timing and lengths. PairFlow is pivoted into host, destination, and URL profiles, and those profiles are merged into a Contextual Summary that is updated across time windows, so the classifier sees the history of a connection, not just one flow. A random forest consumes the Contextual Summary; features such as the mean time difference of sequenced connections (MTDSC), DNS requests per host, destination reached by IP, and resumed connections carry much of the discriminative weight.","core_discovery":"EarlyCrow is a detector for APT malware command and control over HTTP(S). The paper's central claim is that the context surrounding a malicious connection, rather than any single flow feature, separates APT C&C from legitimate web traffic: hosts infected by APTs resolve few domains, keep connections open longer, switch to raw TCP after a camouflage HTTP exchange, contact destinations by IP without DNS resolution, and show timing and byte-ratio patterns distinct from both botnets and benign browsing. The paper encodes that context in PairFlow and Contextual Summaries, and reports that a random forest trained on these summaries detects unseen APT families with a macro F1 of 93.02% and FPR 0.74%, and with macro F1 of 93.72% and accuracy 98.11% in an HTTPS-only mode where HTTP payload features are unavailable.","pith_inferences":["Editorial inference: If the sandbox-to-enterprise transfer holds, EarlyCrow-style contextual summaries could be stacked with domain detectors and SIEMs to catch fallback channels that single-flow detectors miss.","Editorial inference: The TTP-to-feature correlation analysis suggests a testable extension: evaluating whether the same contextual-summary approach detects the raw-TCP C&C cases (Cases III and IV) that the paper leaves to future work.","Editorial inference: Because the paper avoids deep learning due to scarce APT data, a natural next test is whether synthetic augmentation of PairFlow summaries improves generalization further, but this is not shown in the paper.","Editorial inference: The reported HTTPS robustness is partly explained by features that survive encryption, such as byte ratios, timing, DNS request counts, and connection resumption; a reader should check whether these survive real-world CDN and proxy traffic."],"forward_implications":["Deploying EarlyCrow at the network edge should catch APT families it never saw in training, because the held-out evaluation uses malware families excluded entirely from training.","It can flag C&C early, within the first 15 minutes of infection, matching the sandbox observation that C&C contact starts immediately.","HTTPS traffic does not need to be decrypted for detection; the HTTPS-mode classifier keeps macro F1 near 93% without plaintext HTTP features.","The low FPR of 0.74% means few legitimate flows are interrupted, which matters for operational adoption.","PairFlow itself is reusable: it outputs separate JSON variants for FQDN, TCP/UDP/ICMP, HTTP, and HTTPS analysis, not just for this classifier."],"supporting_citations":[{"why":"Supplies the strongest baseline classifier, a web-proxy-based random forest for C&C detection, to which EarlyCrow is compared.","marker":"[39]"},{"why":"Prior APT C&C domain detection work whose threat model and domain features motivate the threat model and FQDN/domain-age features.","marker":"[16]"},{"why":"A contextual flow format that attaches TLS parameters and user agents to flows; PairFlow extends this idea.","marker":"[33]"},{"why":"Provides the causal-analysis definitions (confounders, colliders, data shifts) used to audit and mitigate dataset bias.","marker":"[52]"},{"why":"Supplies the recommended practices for labeling and avoiding bias in security ML datasets, used in dataset construction.","marker":"[51]"},{"why":"Used to troubleshoot the IDS dataset by removing samples without data transfer or failed TCP connections.","marker":"[53]"},{"why":"Source of invariant flow-representation features for detecting unseen malware variants, adapted into PairFlow features.","marker":"[40]"},{"why":"Early-stage APT infection detection via belief propagation on host-domain graphs; provides comparison features for host and domain behavior.","marker":"[13]"},{"why":"NetFlow-based C&C server detection; source of byte statistics and a comparison baseline.","marker":"[10]"}],"fun_headline_variants":["Contextual summaries expose APT C&C traffic unseen in training","93% F1 for APT C&C detection with contextual summaries","EarlyCrow detects APT C&C using context, not just flows","HTTPS-only mode still catches APT C&C at 93.72% F1"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline numbers assume that traffic captured in a public malware sandbox and public repositories behaves like real enterprise APT command-and-control traffic, and the paper does not quantitatively verify that transfer.","fun_headline_variants_meta":{"raw":{"variants":["Contextual summaries expose APT C&C traffic unseen in training","93% F1 for APT C&C detection with contextual summaries","EarlyCrow detects APT C&C using context, not just flows","HTTPS-only mode still catches APT C&C at 93.72% F1"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000681,"raw_usage":{"total_tokens":3092,"prompt_tokens":940,"completion_tokens":2152,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":556,"completion_tokens_details":{"reasoning_tokens":2072}},"tokens_in":556,"tokens_out":2152,"duration_ms":15540,"temperature":1.0,"reasoning_tokens":2072,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T19:36:59.913669+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train EarlyCrow exactly as described, run it on packet captures from a real enterprise network with independently confirmed APT infections and manually verified benign traffic, and compare the macro F1 on families not in training; a material drop below the reported 93%, or a showing that platform-specific fields such as TTL dominate the decisions, would falsify the generalization claim.","supporting_citations":[{"cited_title":"Made: Security analytics for enterprise threat detection,","cited_arxiv_id":null,"evidence_quote":"Supplies the strongest baseline classifier, a web-proxy-based random forest for C&C detection, to which EarlyCrow is compared."},{"cited_title":"H AWK-EYE: Holistic detection of APT command and control domains,","cited_arxiv_id":null,"evidence_quote":"Prior APT C&C domain detection work whose threat model and domain features motivate the threat model and FQDN/domain-age features."},{"cited_title":"Identifying encrypted malware traffic with contextual flow data,","cited_arxiv_id":null,"evidence_quote":"A contextual flow format that attaches TLS parameters and user agents to flows; PairFlow extends this idea."},{"cited_title":"Causality matters in medical imaging,","cited_arxiv_id":null,"evidence_quote":"Provides the causal-analysis definitions (confounders, colliders, data shifts) used to audit and mitigate dataset bias."},{"cited_title":"Dos and don’ts of machine learning in computer security,","cited_arxiv_id":null,"evidence_quote":"Supplies the recommended practices for labeling and avoiding bias in security ML datasets, used in dataset construction."},{"cited_title":"Troubleshooting an intrusion detection dataset: the cicids2017 case study,","cited_arxiv_id":null,"evidence_quote":"Used to troubleshoot the IDS dataset by removing samples without data transfer or failed TCP connections."},{"cited_title":"Optimized invariant representation of network traffic for detecting unseen malware variants,","cited_arxiv_id":null,"evidence_quote":"Source of invariant flow-representation features for detecting unseen malware variants, adapted into PairFlow features."},{"cited_title":"Detection of early-stage enterprise infection by mining large-scale log data,","cited_arxiv_id":null,"evidence_quote":"Early-stage APT infection detection via belief propagation on host-domain graphs; provides comparison features for host and domain behavior."},{"cited_title":"Disclo- sure: detecting botnet command and control servers through large-scale netflow analysis,","cited_arxiv_id":null,"evidence_quote":"NetFlow-based C&C server detection; source of byte statistics and a comparison baseline."}],"review_version":1}