{"id":"e91d1fe8-b742-4b62-a78b-592dbf83281f","arxiv_id":"2411.10279","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":10,"one_line_summary":"LMDetect classifies time-aware subgraphs of authentication events to detect lateral movement, with reported state-of-the-art results on LANL and CERT.","lead":"LMDetect is a graph neural network framework that detects lateral movement in enterprise networks by classifying time-aware subgraphs built from authentication logs. It reports near-perfect detection scores on the LANL and CERT datasets, but the evaluation has unaddressed data leakage risks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Random subgraph split with one-hot ID features creates entity-level leakage; the reported near-perfect results may reflect ID memorization rather than genuine generalization.","rationale":"The reader's weakest assumption correctly identifies the leakage path: random split of overlapping subgraphs from a single HAMG, combined with one-hot ID node features, lets the model memorize malicious entities seen in training and recognize them in test subgraphs. This is the single most load-bearing concern because the paper's headline result — state-of-the-art near-perfect detection — would be an artifact of the evaluation protocol rather than evidence of the proposed time-aware subgraph classification generalizing. The concern is concrete and structurally grounded: Algorithm 1 always includes the event's user, host, and object as core nodes, and Section 5.4's random split places events from the same attacker's campaign into all splits. The proposed entity-disjoint or temporal split would settle it. I also note the secondary issue of selecting the random-walk length K on the same test data in Section 5.8, but the primary problem is the split leakage. Since this aligns with the reader's analysis, and the absence of code or data makes the artifact unverifiable, the rejection verdict should stand; the direction remains reasonable and could be validated with the proposed re-split.","tokens_in":18345,"tokens_out":1883,"duration_ms":20837,"concrete_test":"Recompute the LMDetect row of Table 4 under an entity-disjoint split: assign users, hosts, and servers to train/validation/test partitions so that no entity appears in more than one partition, then sample subgraphs only from events whose entities are in the corresponding partition. If F1 or AUC drops substantially (e.g., below 90 or significantly closer to the GNN baselines), the near-perfect reported values were caused by overlapping one-hot ID leakage. An alternative check is a strict temporal split (train on the first 80% of the time axis, test on the last 20%), which removes future information from training subgraphs.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim is Table 4, where LMDetect achieves F1 99.03 / AUC 99.99 on LANL and F1 98.43 / AUC 99.99 on CERT. Section 5.4 splits 17,400 event samples randomly 6:2:2, and Algorithm 1 step 7 labels each sampled subgraph with the target event's label. Every subgraph is cut from one global HAMG, so subgraphs overlap heavily through shared users, hosts, servers, files, processes, and time-window edges. Because Section 3.2 defines node features as one-hot type plus one-hot ID, a training subgraph containing a malicious user or host provides the model with a direct lookup key for that exact entity. Test subgraphs containing that same entity can then be classified by recognizing its one-hot ID, which is essentially memorization, not lateral-movement generalization. Malicious events in LANL and CERT are tied to a small set of attacker-controlled entities, so the overlap is structurally guaranteed. Random splitting therefore violates the independence premise required for the evaluation to be meaningful. The ablation and sensitivity analyses (§5.6, §5.8) inherit this problem, and Section 5.8's selection of K=32 on the same test data adds a secondary bias by reporting the best-performing hyperparameter configuration. The paper provides no code or data artifacts, so the leakage path cannot be checked post hoc. This does not invalidate the method's direction, but it means the state-of-the-art claim is unsupported as presented.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes LMDetect, a framework for lateral movement detection from authentication logs. It constructs a heterogeneous authentication multigraph (HAMG), samples time-aware subgraphs centered on each authentication event via a time-window and top-k edge selection, and classifies each subgraph with a multi-scale attention encoder combining local attention, relative position encoding based on random walks, and global attention. Experiments on the LANL and CERT datasets report near-perfect detection metrics (e.g., AUC 99.99 on both) and the authors claim state-of-the-art performance over GNN baselines and existing graph-based detectors.","tokens_in":18615,"tokens_out":5777,"duration_ms":51369,"significance":"If the reported results were valid, the paper would make a useful contribution by reframing lateral movement detection as subgraph classification and by introducing a time-aware subgraph generator. The conceptual shift from edge/path classification to time-aware subgraph classification is reasonable and addresses a real limitation of prior GNN-based approaches. However, the evaluation protocol has a serious entity-leakage problem: subgraphs are randomly split while sharing node IDs from a single HAMG, and node features include one-hot IDs. This makes the near-perfect metrics in Table 4 consistent with memorization rather than generalization. The paper also provides no code or data artifacts, so the results cannot be independently checked. The strengths of the idea do not compensate for the unsupported central empirical claim.","major_comments":[{"comment":"The random split of event samples does not prevent subgraphs in the training and test sets from sharing entities. Definition 1 specifies node features as one-hot type concatenated with one-hot ID; Algorithm 1 (step 7) assigns the label of the target event to the entire sampled subgraph. Since all subgraphs are extracted from a single HAMG, a training subgraph containing a malicious entity's ID allows the model to memorize that ID, and a test subgraph containing the same ID can then be classified by recognition of the ID rather than by any lateral-movement behavior. This is a direct label-leakage path. The near-perfect results in Table 4 (e.g., AUC 99.99 on both datasets) are therefore not evidence of generalization, and the claim in Section 5.5 that LMDetect outperforms all methods across all metrics is unsupported.","section":"§5.4, Definition 1, Algorithm 1"},{"comment":"The random-walk length K is selected on the test set. Section 5.8 reports varying K in the set {16, 24, 32, 36, 40} and choosing K = 32 based on the resulting performance metrics; the K = 32 row exactly matches the final Table 4 results. No validation set is used for this hyperparameter choice. Thus the reported performance is an optimistically biased selection of the best test-fold configuration, and the sensitivity analysis does not establish the robustness claimed.","section":"§5.8, Figure 7"},{"comment":"The ablation and sensitivity analyses are computed on the same test folds as the final model, and they inherit the leakage described above. For example, Table 5 reports that removing global attention (LMDetect w/o ge) drops F1 from 99.03 to 87.93 on LANL, but because the evaluation protocol allows ID memorization, the magnitude of these differences is not a reliable measure of the module's contribution. The ablation results cannot support the conclusion that each module significantly enhances detection.","section":"§5.6, Table 5"}],"minor_comments":[{"comment":"The section title contains a typo ('Detecttion' should be 'Detection'), and the text states 'LMDetect performes exceptionally well' ('performes' should be 'performs').","section":"§5.5"},{"comment":"Figures 6 and 7 contain garbled overlaid text and embedded table fragments (e.g., '表格 1' and duplicated 'K TSG k=1 TSG k=2' lines) that make the figures difficult to interpret; the captions and figures should be cleaned up.","section":"Figures 6 and 7"},{"comment":"LMTracker is described inconsistently: Section 2.2 calls it an unsupervised anomalous path detection method using predefined thresholds and meta-paths, while Section 5.2 says it models lateral movement detection as a path classification problem using metapath2vec. The description should be reconciled.","section":"§2.2 and §5.2"},{"comment":"The loss in Eq. (14) is an unweighted cross-entropy, despite the severe class imbalance (17,000 benign vs. 400 malicious samples) described in Section 5.4. The paper should justify this choice or describe any balancing technique used.","section":"§4.3, Eq. (14)"},{"comment":"In Algorithm 1, step 4, the edge feature T is introduced as interaction frequency, but Definition 1 defines an edge feature vector containing interaction type, authentication type, logon type, and authentication orientation; it is not specified how T is combined with these features in the model.","section":"Algorithm 1, Definition 1"},{"comment":"The relative position encoding concatenates P_{i,i} (self-transition probabilities after random walks) to node features, which seems uninformative; the authors should clarify whether node features should instead aggregate transition probabilities to or from other nodes.","section":"§4.2.2, Eq. (7)"},{"comment":"The paper states '17,000 benign event samples and 400 malicious event samples on both datasets' but does not explain how these samples are selected or whether the 400 malicious events constitute the complete set of malicious events in each dataset.","section":"§5.4"},{"comment":"No code or data artifacts are provided, which prevents post-hoc verification of the experimental results and the claimed leakage-free evaluation.","section":"General"}],"recommendation":"reject","confidential_remarks":"The paper's direction is promising, but the evaluation is invalid as presented. A major revision would need to re-run experiments with a time-based or entity-disjoint split, remove one-hot IDs from node features, and report validation-based hyperparameter selection. Given the scope of rework required and the absence of artifacts, I cannot recommend acceptance. I would not rule out a future submission that addresses these issues."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: LMDetect proposes a genuinely different framing for lateral movement detection—time-aware subgraph classification instead of edge or path classification—and the components are reasonable. But the empirical section does not support the near-perfect numbers. The random split of overlapping subgraphs from one shared graph, combined with one-hot ID node features, lets the model recognize specific users and hosts from training in test subgraphs. That is entity-level leakage, not generalization. And K=32 is selected by sweeping on the same test folds. So the headline results are not believable as stated.\n\nWhat is actually new: the task formulation. Prior work (LMTracker, Euler, Latte, HOLMES) mostly treats lateral movement as edge classification, path classification, or link prediction. Modeling each authentication event as a labeled subgraph with a time window and top-k edge sampling is a coherent idea, and it has practical scalability appeal. The multi-scale encoder—local GAT-style attention plus a graph transformer with random-walk positional features—is a combination of known pieces, but applied in a sensible way. The ablation at least shows the time-aware subgraph helps on CERT, which is a useful sanity check.\n\nSoft spots, in order of severity. First, the leakage path. Section 3.2 defines node features as one-hot type plus one-hot ID; Algorithm 1 labels each subgraph with the target event's label; Section 5.4 splits the samples randomly. Because all subgraphs come from one global HAMG, malicious training and test subgraphs share the same attacker-controlled users, hosts, and files. A model with access to one-hot IDs can effectively memorize those entities. The near-perfect LANL scores (F1 99.03, AUC 99.99) are exactly what you would expect from such memorization. The paper needs an entity-disjoint or temporal split, or at least feature embeddings that do not include raw IDs. Second, K is tuned on the same data and reported as the final configuration; that is fitting to the benchmark rather than an honest estimate. Third, no code or data artifacts are provided, so the results cannot be checked.\n\nI want to be fair: the direction is reasonable, and the authors are not trying to deceive—they used a standard random split without realizing the graph structure makes it invalid. That said, the central empirical claim is unsupported as presented. This paper is for a security audience that wants to see the idea; it needs major revision before the numbers can be trusted. I would still send it to review—the task is relevant and the formulation is worth engaging with—but a reviewer should focus on the evaluation protocol and require a clean split plus code. Not a paper I would cite in the next year.","headline":"Plausible new framing for lateral movement detection, but the near-perfect results are likely an artifact of entity leakage from random subgraph splits with one-hot ID features.","tokens_in":19202,"tokens_out":2833,"would_cite":false,"duration_ms":27214,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Lateral movement shows up as small time-aware subgraphs in authentication logs, and a classifier on those subgraphs reports F1 above 98% on two public corpora.","keywords":["lateral movement detection","authentication logs","heterogeneous multigraph","subgraph classification","graph attention network","time-aware sampling","anomaly detection","advanced persistent threat"],"falsifier":"Retrain the model on events from the first half of each corpus and test on events from the second half, or split by disjoint sets of users and hosts, and compare precision, recall, and AUC; a large drop from the reported ~99% F1 and ~99.99% AUC would indicate the random-split results relied on shared entities between training and test subgraphs.","tokens_in":18065,"feed_emoji":"🛡️","tokens_out":7436,"duration_ms":66772,"temperature":0.7,"pith_summary":"Lateral movement is the stage of an advanced attack where an intruder hops between machines using ordinary-looking logins, so it tends to hide in the noise of authentication logs. This paper argues that the right unit of analysis is not a single login event but the small, time-bounded subgraph of related activity around it. Its framework, LMDetect, builds a heterogeneous multigraph from authentication logs, samples a time-aware subgraph for each event, and classifies that subgraph as benign or malicious using a multi-scale attention encoder. On two public authentication-log corpora the framework reports F1 scores above 98% and AUC near 99.99%, outperforming every comparison method on every reported metric. The authors argue this makes subgraph classification a practical, scalable way to catch multi-step intrusions from log data enterprises already collect.","feed_headline":"Authentication subgraphs catch lateral movement at 99% F1","feed_subtitle":"Graph model flags multi-step intrusions from login logs, scoring near-perfect precision on two real-world test sets.","key_machinery":"The load-bearing object is the time-aware authentication subgraph produced by the paper's subgraph generator. Starting from the target event, it unions the 1-hop neighbors of its user, device, and object nodes, filters edges to a window $[t_i-\\tau, t_i+\\tau]$, collapses parallel edges into an interaction-frequency feature $T$, and keeps the top-$k$ edges between core and auxiliary nodes, which makes each training instance small enough for mini-batch learning. The second piece is the multi-scale attention encoder: a local attention layer, a relative-position encoding built from powers $M^K$ of the random-walk transition matrix $M=D^{-1}A$, and a global attention transformer, whose fused local and global features are pooled and classified. The relative-position encoding is what lets the model see long-range dependencies inside a small subgraph, and the ablation study credits the global attention module with much of the performance.","core_discovery":"The paper's central claim is that lateral movement is visible as a local, time-aware structure rather than as isolated edges: around each authentication event, the one-hop neighborhood pruned to a short time window and to the most frequent interactions contains enough signal to label the event as benign or malicious. LMDetect makes this concrete by constructing a Heterogeneous Authentication Multigraph (HAMG) in which users, hosts, servers, files, and processes are nodes and login, connection, access, and creation interactions are edges with timestamps. For each target event $z_i=\\langle t_i, U_i, D_i, O_i\\rangle$, a generator merges the 1-hop neighborhoods of the three core entities, removes edges with $|t-t_i|>\\tau$, aggregates parallel edges by interaction frequency, and keeps the top-$k$ edges by frequency; the label of the event is assigned to the resulting subgraph. A multi-scale attention encoder, combining local attention, random-walk relative-position encoding, and global attention, produces a sum-pooled subgraph embedding that a softmax head classifies. The paper reports that this design outperforms all compared methods on both datasets across all metrics, with near-perfect scores on the larger corpus.","pith_inferences":["Editorial inference: the reported near-perfect numbers may partly reflect memorization of entity IDs rather than structural generalization, because the random 6:2:2 split lets training and test subgraphs share users, hosts, and files; a temporal or entity-disjoint split would test this.","Editorial inference: the window length and top-$k$ are tuned per corpus (one hour versus three hours), so deployment would likely require per-network calibration rather than a fixed setting.","Editorial inference: the same event-centered time-aware subgraph recipe could transfer to other security questions, such as detecting data exfiltration, malicious file access, or insider-threat actions that also show up as small contextual subgraphs around a logged event."],"forward_implications":["Lateral movement detection can be cast as subgraph classification of authentication events, and on the two corpora this paradigm beats the compared edge-, path-, and link-prediction approaches.","Time-aware subgraph sampling is itself a source of signal: graph-convolution, graph-attention, and sampling-aggregation baselines all improve when they receive time-aware subgraphs instead of randomly sampled ones.","The global attention module is the main driver of performance; removing it drops F1 from 99.03 to 87.93 on one corpus and from 98.44 to 79.87 on the other.","One-hop neighborhoods suffice and two-hop neighborhoods hurt, suggesting that direct, temporally nearby interactions carry the discriminative information for lateral movement.","Because subgraphs are small and trained in mini-batches, the framework is designed to scale to the volume of real enterprise authentication logs."],"supporting_citations":[{"why":"Supplies the larger public authentication-log corpus used to train and evaluate the framework.","marker":"[29]"},{"why":"Supplies the second public corpus with labeled insider-threat and lateral-movement events used for evaluation.","marker":"[30]"},{"why":"Provides the graph convolutional baseline whose subgraph-classification performance LMDetect is compared against.","marker":"[31]"},{"why":"Provides the graph attention baseline used in the comparison and ablation.","marker":"[32]"},{"why":"Provides the sampling-based aggregation baseline that performs best among the non-time-aware graph methods.","marker":"[33]"},{"why":"Supplies LMTracker, the heterogeneous-graph path-classification method used as a comparison baseline.","marker":"[20]"},{"why":"Supplies Euler, the temporal graph link-prediction method used as a comparison baseline.","marker":"[19]"},{"why":"Provides the metapath2vec embeddings on which the LMTracker baseline builds its path representations.","marker":"[34]"}],"fun_headline_variants":["Time-aware subgraphs reveal lateral movement in login logs","Lateral movement caught by local graph patterns","Authentication logs expose stealthy intrusions via subgraph analysis","Graph-based method detects lateral movement with near-perfect F1","Subgraph classifier flags lateral movement at 99% F1"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that training and test subgraphs are independent, even though all of them are cut from one shared authentication graph and overlap in users, hosts, files, and time; if the model memorizes entity identities during training, the near-perfect test scores would not show real generalization.","fun_headline_variants_meta":{"raw":{"variants":["Time-aware subgraphs reveal lateral movement in login logs","Lateral movement caught by local graph patterns","Authentication logs expose stealthy intrusions via subgraph analysis","Graph-based method detects lateral movement with near-perfect F1","Subgraph classifier flags lateral movement at 99% F1"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000634,"raw_usage":{"total_tokens":2949,"prompt_tokens":992,"completion_tokens":1957,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":608,"completion_tokens_details":{"reasoning_tokens":1888}},"tokens_in":608,"tokens_out":1957,"duration_ms":13286,"temperature":1.0,"reasoning_tokens":1888,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T19:47:08.714995+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the model on events from the first half of each corpus and test on events from the second half, or split by disjoint sets of users and hosts, and compare precision, recall, and AUC; a large drop from the reported ~99% F1 and ~99.99% AUC would indicate the random-split results relied on shared entities between training and test subgraphs.","supporting_citations":[{"cited_title":"Cyber security data sources for dynamic network re- search,","cited_arxiv_id":null,"evidence_quote":"Supplies the larger public authentication-log corpus used to train and evaluate the framework."},{"cited_title":"Insider Threat Test Dataset,","cited_arxiv_id":null,"evidence_quote":"Supplies the second public corpus with labeled insider-threat and lateral-movement events used for evaluation."},{"cited_title":"Graph attention networks,","cited_arxiv_id":null,"evidence_quote":"Provides the graph attention baseline used in the comparison and ablation."},{"cited_title":"Lmtracker: Lateral movement path detection based on heterogeneous graph embedding,","cited_arxiv_id":null,"evidence_quote":"Supplies LMTracker, the heterogeneous-graph path-classification method used as a comparison baseline."},{"cited_title":"Euler: Detecting network lateral movement via scalable temporal link prediction,","cited_arxiv_id":null,"evidence_quote":"Supplies Euler, the temporal graph link-prediction method used as a comparison baseline."},{"cited_title":"metapath2vec: Scalable representation learning for heterogeneous networks,","cited_arxiv_id":null,"evidence_quote":"Provides the metapath2vec embeddings on which the LMTracker baseline builds its path representations."}],"review_version":1}