{"id":"23534e7f-8bd0-4f85-94a6-c14b79246aed","arxiv_id":"2501.02796","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"GraphDART applies graph distillation to provenance graphs, training a GNN on a condensed benign graph to flag malicious nodes at a fraction of the original graph's size.","lead":"GraphDART compresses large system-event graphs down to a fraction of their original size, then trains a graph neural network to flag nodes that are unusual and may indicate advanced persistent threats. The paper reports detection accuracy close to a state-of-the-art baseline while making GNN training much faster, which could make deep-learning-based threat detection practical on very large systems.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central '5% graph with comparable detection' claim is unverifiable because Section V-B4 never defines how the GNN's per-node type predictions are converted into the binary malicious/benign decisions used to compute precision, recall, and F1.","rationale":"The reader's weakest assumption points at the same gap I would flag: the paper never specifies how the GNN's node-type predictions become binary malicious/benign flags. I considered the omitted distillation overhead and the unexplained FLASH replication discrepancy as alternative concerns. Both are real issues, and the distillation overhead is directly relevant to the efficiency contribution. However, they are secondary to the decision-rule problem: if the detection metrics themselves are not well-defined, then neither the accuracy-efficiency trade-off nor the comparison with FLASH can be assessed. The decision-rule gap is therefore the single most load-bearing concern, and the reader's REJECT verdict with a path toward conditional acceptance is appropriate. I would not change the verdict, hence UNCHANGED.","tokens_in":18429,"tokens_out":6091,"duration_ms":63384,"concrete_test":"Obtain the authors' evaluation code and identify the exact function that maps per-node type predictions (or confidence scores) to the TP/FP/FN/TN counts underlying Tables III–V. Fix a decision rule before any test-set inspection, for example: flag a test node as malicious iff its predicted type differs from the type predicted by a model trained on benign validation data alone, or if its prediction confidence falls below a threshold calibrated on benign validation data. Then recompute Table IV, row r=0.006, GraphDART_gcdm; if the reported F1 of 0.78 cannot be reproduced from that out-of-sample decision rule, or if no such rule exists in the released pipeline, the central claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's strongest claim is that a distilled graph of 5% or less of the original provenance graph yields detection performance comparable to FLASH. To evaluate that claim, the precision, recall, and F1 numbers in Tables III–V must correspond to well-defined out-of-sample detection decisions. The paper stops short of defining that mapping. Section V-B3 makes node types the labels Y_PG; Section V-B4 says the GNN learns to predict types of benign nodes and 'misclassifies malicious ones'; Section VI-C says 'the prediction of the trained GNN for test input graph data T′ are evaluated against the actual node labels in the ground truth.' No sentence defines the decision rule that turns a predicted node type (or prediction confidence) into a positive or negative detection, nor any threshold. Because training is benign-only, the detector is an anomaly detector, and an anomaly detector is only well-defined with an explicit scoring and thresholding step. Without that step, the reported metrics, e.g., Theia r=0.006 GraphDART_gcdm precision 0.64, recall 1.0, F1 0.78, cannot be reproduced or interpreted as out-of-sample results; they could reflect a threshold chosen after inspecting test labels. Since the efficiency benefit matters only if the detection-quality claim is valid, the missing decision rule is the load-bearing weakness.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"GraphDART proposes a modular graph-distillation framework for APT detection on provenance graphs. The pipeline constructs a provenance graph from system logs, develops node features following FLASH, applies one of six distillation methods (random, herding, kcenter, GCDM, GCond, SGDD) to condense the benign training graph, and then trains a GNN on the condensed graph to detect malicious nodes by misclassification of their benign node types. The paper evaluates detection accuracy, precision, recall, F1, and GNN training time on the DARPA TC E3 Cadets, Theia, and FiveDirections datasets, and claims that GraphDART matches or exceeds FLASH while using a condensed graph of 5% or less of the original size.","tokens_in":18654,"tokens_out":4051,"duration_ms":42803,"significance":"If the empirical claims hold, the paper would make a useful contribution: it is, to my knowledge, the first systematic application of graph distillation to provenance-graph APT detection, and the modular treatment of several distillation methods is a practical strength. The benign-only distillation strategy is an interesting idea that could reduce the cost of GNN-based detectors. The paper also reports training-time savings on public datasets and gives a reasonably detailed description of the experimental environment. However, the central efficiency-accuracy claim currently rests on an undefined detection decision rule, on an ad hoc removal of rare node classes, and on an incomplete accounting of runtime, so the reported metrics are not yet interpretable as evidence for the claimed trade-off.","major_comments":[{"comment":"The paper never defines the decision rule that maps the GNN's per-node type predictions to the binary malicious/benign decisions used to compute TP, FP, FN, and TN. Section V-B4 says the GNN learns benign node types and 'misclassifies malicious ones,' and Section VI-C says predictions are 'evaluated against the actual node labels in the ground truth,' but no sentence specifies whether a node is flagged malicious when the predicted type differs from the ground-truth type, when the softmax confidence is below a threshold, or via some other scoring function. Since the model is trained on benign data only, the detector is an anomaly detector, and an anomaly detector is well-defined only with an explicit scoring and thresholding step. Without this step, the precision, recall, and F1 values in Tables III-V (e.g., GraphDART_gcdm at r=0.006 on Cadets: precision 0.58, recall 1.00, F1 0.73) are not reproducible, and if the rule was chosen after inspecting test labels, they are not out-of-sample metrics. This is load-bearing because the central claim of 'comparable or better performance at 5% or less graph size' cannot be assessed without well-defined detection decisions.","section":"V-B4 and VI-C"},{"comment":"The paper removes all node classes that account for less than 1% of nodes before graph distillation, calling them outliers, but provides no evidence that these classes are irrelevant to APT detection. In FiveDirections, for example, classes 1, 3, 4, 5, and 7 are removed, including FILE OBJECT CHAR, SRCSINK DATABASE, FILE OBJECT UNIX SOCKET, FILE OBJECT BLOCK, and SRCSINK PROCESS MANAGEMENT; some of these could plausibly carry attack-relevant activity. Since the stated goal is to preserve 'essential structural and contextual information' for detecting malicious nodes, the 1% threshold is an ad hoc free parameter, and the paper does not report whether any malicious nodes in the ground truth belong to removed classes, nor does it provide a sensitivity analysis over this threshold. This omission directly affects the validity of the claim that the distilled graph retains the information needed for APT detection.","section":"V-B3 and Table II"},{"comment":"The reported training time t covers only the GNN training phase on the condensed graph, not the time spent on graph distillation itself. The paper's efficiency argument, however, is about the total computational overhead of GraphDART relative to processing the original provenance graph. For Theia, FLASH takes 282 seconds for GNN training while GraphDART takes 6-12 seconds, but if the distillation step (e.g., GCDM or SGDD on a graph with 300k nodes and 9M edges) takes substantially longer, the end-to-end saving may be much smaller or even negative. The paper should report the distillation time or total end-to-end time, at least for the reduction rates used in the main comparisons.","section":"VI-D and Tables III-V"},{"comment":"The FLASH baseline comparison is weakened by the paper's own admission that it could not reproduce FLASH's original results. Section V-B4 states that 'some discrepancies in the results were observed and we were unable to replicate exactly the original results presented in FLASH,' but no details are given about the reproduced FLASH configuration, the magnitude of the discrepancies, or how the FLASH numbers in Figure 4 and Tables III-V were obtained. Since FLASH is the sole detection baseline and the paper's headline claim is 'comparable to FLASH,' the reader needs to know which FLASH results are being compared against and whether the discrepancy affects the comparison. At minimum, the reproduced FLASH metric values should be reported alongside the original published values.","section":"VI-C"}],"minor_comments":[{"comment":"There is a typo: 'accounts for less that 1%' should read 'accounts for less than 1% of the total nodes.'","section":"V-B3"},{"comment":"The paper says 'Labels from ThreaTrace [49] are utilized as ground truth' but does not explain how ThreaTrace's labels are mapped to the binary malicious/benign distinction used in the metrics. A precise mapping would improve reproducibility.","section":"VI-A"},{"comment":"The caption says 'Average results with r ∈ {0.006, 0.004, 0.002}' but it is not clear whether the averages are over reduction rates for each variant or over repeated runs; the main text also does not report variance across the nondeterministic coreset methods.","section":"Figure 4"},{"comment":"Many cells for the random, herding, and kcenter variants are marked '-' without explanation; the paper should state whether these configurations were not run or failed, and if not run, why the comparison is incomplete.","section":"Tables III-V"},{"comment":"The word 'bellow' should be 'below' in the sentence introducing the performance-matching objective.","section":"Equation (4)"},{"comment":"The FLASH reference [27] lists page numbers as 'pp. 139-139,' which appears to be a truncated or erroneous page range and should be corrected.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper fits the journal's scope and addresses a timely problem, but the evaluation section is currently missing a load-bearing specification: the binary decision rule for anomaly detection. This is fixable if the authors can provide the exact scoring/threshold procedure and confirm it was chosen without peeking at test labels. I would also want to see the distribution of malicious nodes across the removed rare classes and an end-to-end runtime comparison that includes distillation time. If the authors cannot supply the decision rule or demonstrate that malicious nodes are not systematically removed by the 1% class threshold, the paper would need to be rejected rather than revised."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take on GraphDART. The genuinely new thing is applying graph distillation to provenance graphs for APT detection, and the paper does it in a modular way—six distillation methods, three DARPA E3 datasets, and a clean integration with FLASH's GNN. If the efficiency claim holds, a 95% graph-size reduction with roughly comparable F1 is a meaningful operational result for anyone trying to run GNN-based detection on large provenance graphs. The authors also get credit for trying to reuse public code (GC-Bench, FLASH) and for acknowledging the FLASH replication gap rather than papering over it.\n\nThe soft spots are real, though, and they cluster in the evaluation. The biggest one is exactly what your stress-test flags: Section V-B4 says the GNN predicts node types and that malicious nodes are 'misclassified,' and Section VI-C says predictions are 'evaluated against the actual node labels,' but nowhere is the binary decision rule defined. Does a node count as malicious whenever predicted type != true type? Is there a threshold on the prediction probability or entropy? The paper says 'as in FLASH,' but FLASH's exact flagging rule isn't reproduced in the text. That's load-bearing because precision, recall, and F1 in Tables III–V are uninterpretable without it. It's also trivially fixable: state the rule in one sentence.\n\nThe other issues are proportionally less severe. Removing classes with <1% of nodes before distillation could bias the detection task, and the paper doesn't analyze what happens to benign nodes of those rare types at inference. Distillation time itself is never reported; the reported 't' is only GNN training time, so the total pipeline cost is unknown. There are no variance estimates despite acknowledged nondeterminism in the coreset methods. And the promised 'Details will be provided in the next section' about FLASH replication discrepancies never materializes—Section VI has no such details.\n\nThe central argument is not broken; it's under-specified. I'd send this to serious review and ask for a major revision that adds the decision rule, reports distillation cost and variance, and addresses the class-removal impact. It's a useful paper for the provenance-graph and security-ML community, and the first-application novelty plus the modular comparison are enough to justify referee time.","headline":"First graph-distillation-for-APT-detection paper with a real efficiency claim, but the missing decision rule for anomaly flags makes the headline numbers hard to trust as reported.","tokens_in":19242,"tokens_out":3835,"would_cite":true,"duration_ms":34830,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"GraphDART claims that graph distillation can shrink provenance graphs to 5% or less of their original size while preserving enough structure for GNN-based APT detection to match or beat full-graph baselines.","keywords":["graph distillation","provenance graph","advanced persistent threat","APT detection","graph neural network","anomaly detection","node classification","cyber-physical-social systems"],"falsifier":"Run GraphDART with a pre-registered decision rule—for example, flag any node whose predicted type differs from its benign type, with ties broken by a fixed confidence threshold chosen on training data only—and recompute the metrics on the three benchmark test splits. If the precision and recall numbers in the tables cannot be reproduced without post hoc threshold selection, or if the distilled-graph model fails to flag a held-out attack campaign, the 5%-size claim is falsified.","tokens_in":18188,"feed_emoji":"🛡️","tokens_out":6218,"duration_ms":56563,"temperature":0.7,"pith_summary":"GraphDART is a framework that applies graph distillation to provenance graphs—the directed graphs built from system logs that trace which process touched which file—so that APT detection can run on a tiny synthetic graph instead of the original. The paper claims that a distilled graph containing 5% or fewer of the original nodes still carries enough structural and contextual information for a graph neural network to detect malicious nodes as reliably as training on the full graph. If true, this would remove the main scalability bottleneck for GNN-based intrusion detection, cutting training time on large audit logs from minutes to seconds. The framework is modular: six different distillation methods (coreset selection, distribution matching, and gradient matching) can be plugged in and compared.","feed_headline":"APT detection on a 5% graph matches full-size results","feed_subtitle":"GraphDART distills provenance graphs down to 5% of nodes and still flags attacks with near-perfect recall.","key_machinery":"The load-bearing component is the graph distillation module, which takes the provenance graph (nodes, adjacency, node features, node-type labels) and produces a condensed graph $S_{PG}$ whose size is controlled by a reduction rate $r$ ($r \\le 0.01$ for the main experiments). It wraps six distillation algorithms: random, herding, and kcenter coreset selection; GCDM (receptive-field distribution matching); and GCond and SGDD (gradient matching). Node features are built like the full-graph baseline: Word2Vec embeddings plus positional encoding of a node's attributes and the types of its timestamp-sorted incident edges. Classes with fewer than 1% of nodes are dropped before distillation, and the condensed benign graph trains the GNN for node-type classification; malicious nodes are detected as classification failures.","core_discovery":"The central discovery is that provenance graphs can be distilled—condensed into a synthetic graph via optimization—so that a GNN trained only on benign distilled nodes still flags malicious nodes through misclassification: the distilled graph is synthesized exclusively from benign system activity, and at inference malicious nodes deviate from the learned benign neighborhood patterns and are misclassified, with that misclassification serving as the detection signal. On three benchmark provenance-graph datasets, GraphDART with gradient-matching and distribution-matching distillation achieves accuracy and F1 comparable to the full-graph GNN baseline, with near-perfect recall, while the smaller graphs reduce GNN training time from minutes to a few seconds on the largest dataset.","pith_inferences":["If the central trade-off holds, graph distillation could be applied not just to training but to streaming inference-time provenance graphs, enabling real-time APT detection without retaining full audit history.","The benign-only strategy suggests a data-augmentation view: distillation may act as a form of regularization, and the 1% class-drop heuristic could be replaced by a learned class-weighting scheme to preserve rare node types that might matter for stealthy attacks.","The 5% claim is tied to the specific feature representation (Word2Vec plus positional encoding); a testable prediction is that richer node embeddings would preserve detection quality at even lower reduction rates.","The near-perfect recall across methods hints that almost any node deviating from benign structure is flagged, so the real bottleneck is precision; future work should focus on scoring suspicious nodes rather than binary misclassification."],"forward_implications":["GNN-based APT detection can be trained on provenance graphs reduced to 5% or less of the original node count with detection performance comparable to full-graph training.","Training time on large datasets drops dramatically: on the largest dataset, from 282 seconds with the full graph to about 6–9 seconds with distilled graphs, with the gap widening as training epochs increase.","The framework's modularity means the distillation method can be selected per dataset; gradient-matching and distribution-matching methods give stable results, while coreset methods fluctuate.","The benign-only distillation strategy removes the need for labeled attack data during distillation, since the distilled graph is synthesized from benign logs alone and attacks are detected by deviation at inference.","Recall stays near-perfect across all datasets, while precision is the limiting factor; improvements in false-positive control would directly raise F1."],"supporting_citations":[{"why":"Supplies the GNN architecture, node-feature construction (Word2Vec plus positional encoding), and the full-graph baseline that GraphDART is compared against.","marker":"[27]"},{"why":"Provides the three public benchmark provenance-graph datasets (Cadets, Theia, Fivedirections) on which all methods are evaluated.","marker":"[12]"},{"why":"Provides the ground-truth node labels used to evaluate APT detection performance.","marker":"[49]"},{"why":"The GCond gradient-matching graph distillation method integrated as a GraphDART variant.","marker":"[30]"},{"why":"The SGDD gradient-matching graph distillation method, also a GraphDART variant.","marker":"[38]"},{"why":"The GCDM distribution-matching distillation method based on receptive field distribution matching, another GraphDART variant.","marker":"[31]"},{"why":"Word2Vec is used to embed text-based node features into dense vectors.","marker":"[28]"},{"why":"Provides the random coreset selection method and the benchmark implementations used for the distillation module.","marker":"[48]"}],"fun_headline_variants":["GraphDART shrinks APT graphs to 5%, matching full-graph detection","Distill provenance graphs to 5% and still catch APTs fast","APT detection via misclassification on tiny distilled graphs","Train on benign distilled nodes, flag APTs in seconds","GraphDART: 5% of the graph, near-perfect APT recall"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported precision and recall rest on an unspecified rule for turning the GNN's per-node type predictions into a malicious/benign verdict; if that rule is tuned after looking at test labels, the central efficiency-accuracy trade-off is not demonstrated out-of-sample.","fun_headline_variants_meta":{"raw":{"variants":["GraphDART shrinks APT graphs to 5%, matching full-graph detection","Distill provenance graphs to 5% and still catch APTs fast","APT detection via misclassification on tiny distilled graphs","Train on benign distilled nodes, flag APTs in seconds","GraphDART: 5% of the graph, near-perfect APT recall"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000203,"raw_usage":{"total_tokens":1352,"prompt_tokens":876,"completion_tokens":476,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":492,"completion_tokens_details":{"reasoning_tokens":382}},"tokens_in":492,"tokens_out":476,"duration_ms":5241,"temperature":1.0,"reasoning_tokens":382,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:04:13.017590+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run GraphDART with a pre-registered decision rule—for example, flag any node whose predicted type differs from its benign type, with ties broken by a fixed confidence threshold chosen on training data only—and recompute the metrics on the three benchmark test splits. If the precision and recall numbers in the tables cannot be reproduced without post hoc threshold selection, or if the distilled-graph model fails to flag a held-out attack campaign, the 5%-size claim is falsified.","supporting_citations":[{"cited_title":"Flash: A comprehensive approach to intrusion detection via provenance graph representation learning,","cited_arxiv_id":null,"evidence_quote":"Supplies the GNN architecture, node-feature construction (Word2Vec plus positional encoding), and the full-graph baseline that GraphDART is compared against."},{"cited_title":"Darpa engagement 3","cited_arxiv_id":null,"evidence_quote":"Provides the three public benchmark provenance-graph datasets (Cadets, Theia, Fivedirections) on which all methods are evaluated."},{"cited_title":"Threatrace: Detecting and tracing host-based threats in node level through provenance graph learning,","cited_arxiv_id":null,"evidence_quote":"Provides the ground-truth node labels used to evaluate APT detection performance."},{"cited_title":"Graph condensation for graph neural networks,","cited_arxiv_id":null,"evidence_quote":"The GCond gradient-matching graph distillation method integrated as a GraphDART variant."},{"cited_title":"Does graph distillation see like vision dataset counterpart?,","cited_arxiv_id":null,"evidence_quote":"The SGDD gradient-matching graph distillation method, also a GraphDART variant."},{"cited_title":"Efficient estimation of word representations in vector space,","cited_arxiv_id":null,"evidence_quote":"Word2Vec is used to embed text-based node features into dense vectors."},{"cited_title":"Graph condensation benchmark (gc-bench)","cited_arxiv_id":null,"evidence_quote":"Provides the random coreset selection method and the benchmark implementations used for the distillation module."}],"review_version":1}