{"id":"f5b4366d-7175-4528-84db-125897411799","arxiv_id":"2501.16393","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A knowledge-graph plus weighted imbalanced learning plus LLM multi-agent framework reports a 3-4% improvement in insider-threat capture on the CERT dataset.","lead":"This paper combines a knowledge graph, weighted imbalanced learning, and LLM agents to flag insider threats, reporting a 3-4% gain in threat capture over one unweighted baseline on the CERT benchmark. It is a readable early-stage example of adding LLM explanations onto structured graph analytics, but the evaluation is preliminary and lacks baselines.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. 2's unconstrained per-sample weights are not a well-posed objective; as printed they would be driven to zero for positive samples, so the method behind Table 1 is underspecified.","rationale":"I read the paper as a preliminary system proposal whose central claim is '3–4% better threat capture plus natural-language interpretation.' The only numerical evidence is Table 1, comparing a GBM with and without weights learned via Eq. 2. The weakest link is not the overall architecture but the training objective of the supposedly beneficial component. As written, Eq. 2 is not a proper objective in λ: for positive samples the optimal λ is at a boundary (zero) or unbounded, depending on allowed sign. This is a stronger defect than the reader's 'could overfit the validation split': the procedure is not merely at risk of overfitting, it is underspecified to the point that the reported numbers cannot be reproduced from the paper alone. I also note that Theorem 1 addresses fixed α weights, not the learned per-sample λ_i, so the theoretical section does not vouch for the method. The graph-similarity bound and the Jaccard analysis in §4.3.2 are correct but tangential to Table 1. Given the public CERT dataset and the concrete demo, the paper has a useful starting point; the correct response is to require the authors to specify the constrained/penalized λ objective and provide cross-validated results. This keeps the reader's CONDITIONAL verdict rather than moving to acceptance.","tokens_in":11101,"tokens_out":7543,"duration_ms":71134,"concrete_test":"Run an independent implementation of the exact λ_i learning method from the authors or the cited [Zhang et al., 2022] reference. First check whether Eq. 2 as printed has a finite, non-degenerate optimum for λ; then re-estimate Model 2 under the actual constraint/penalty on the same 70/30 CERT split with 5-fold cross-validation. If the printed objective is degenerate, or if the top-3% gain (60% vs 56%) does not survive cross-validation, the central improvement claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Table 1 (§5.2.3) is the only quantitative support for the claimed 3–4% improvement, and it is credited to 'learnable weights' λ_i from Eq. 2. As printed, Eq. 2 is min_{β,λ} −Σ_i[λ_i y_i log π(β^T x_i)+(1−y_i)log(1−π(β^T x_i))], with no constraint or penalty on λ. For a positive sample, y_i=1 and π_i=π(β^T x_i) ∈ (0,1), so the term is −λ_i log π_i > 0; minimizing over λ_i drives it to 0 (or to −∞ if λ_i is allowed to be negative). Hence the printed objective does not define an 'upweighting' procedure—it is degenerate unless a constraint or regularizer is added, and none is stated. The self-cited method [Zhang et al., 2022] is not reproduced, and Theorem 1 (§4.3.1) treats a fixed class weight α>1, not per-sample learned λ_i. The empirical claim therefore rests on an underspecified training procedure; the reported 60% vs 56% / 98% vs 95% gap cannot be reproduced from the paper as written and may be an artifact of the omitted λ regularization rather than a genuine imbalanced-learning gain.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a multi-agent framework for network threat detection that combines a user-activity knowledge graph, an imbalanced-learning-weighted classifier, and a large language model for retrieval and interpretation. The framework is applied to the CERT insider-threat dataset, and the authors report that the weighted Gradient Boosting model captures 3-4% more true threats at fixed false-positive operating points (Table 1) and that an LLM-based demo provides natural-language risk explanations. The paper also contains three theoretical components: a consistency result for weighted logistic regression (Theorem 1), a bound on the weighted Jaccard similarity under graph updates (Theorem 2), and a cooperative-game core result for LLM collaboration (Theorem 3). The central empirical claim is that learning per-sample weights via Equation (2) improves threat-capture rate over an unweighted model.","tokens_in":1742,"tokens_out":1856,"duration_ms":50648,"significance":"If the claimed 3-4% detection improvement were properly supported, the imbalanced-learning weighting component would be a modest but practically meaningful contribution to insider-threat detection, and the LLM interpretation layer addresses a real usability gap. The use of a public benchmark (CERT) is a strength, as are the explicit theoretical bounds in Theorems 2 and 3. However, the empirical support is currently preliminary: Table 1 rests on a single 70/30 split with no error bars, no cross-validation, and only one baseline; the LLM interpretation is demonstrated on substituted online documents rather than the CERT content; and the printed learning objective in Equation (2) is degenerate. These issues are load-bearing for the paper's central claims, so the contribution is not yet established in reproducible form.","major_comments":[{"comment":"The objective in Equation (2) as printed is degenerate and does not define the claimed 'learnable weights' procedure. For a positive sample (y_i = 1), the term is −λ_i log π(β^T x_i), and since log π(β^T x_i) is negative, minimizing over λ_i drives λ_i to zero (or to −∞ if λ_i is unrestricted); no constraint or penalty on λ_i is stated anywhere. This means the 4% and 3% gains in Table 1 cannot be reproduced from the manuscript as written. The reference to [Zhang et al., 2022] does not supply the missing specification, and Theorem 1 uses a fixed class weight α > 1 rather than per-sample learned λ_i, so it does not justify the procedure. The authors must state the actual constrained/regularized objective (or reproduce the exact optimization from the cited paper) and confirm that Table 1 was generated with that objective.","section":"§4.3, Eq. (2); §5.2.3, Table 1"},{"comment":"The quantitative comparison is based on a single 70/30 train-validation split, with no error bars, no repeated runs, no cross-validation, and no ablations. The paper's own Future Work section lists 'Cross-validation experiments' as a future step, which confirms that the reported numbers are not backed by variance estimates. The 3-4% gain could be an artifact of a favorable split or of the unregularized weight-fitting procedure in Equation (2). To support the central claim, the authors should provide at least multiple random splits or cross-validated estimates, and compare against a standard cost-sensitive baseline (e.g., fixed class weights) in addition to the unweighted model.","section":"§5.2.3, Table 1; §6 Future Work"},{"comment":"The LLM interpretation claim is not demonstrated on the CERT dataset used for the quantitative evaluation. The text explicitly states that the CERT content produced incoherent responses and that 'for demonstration purposes, we used the online public documents to build the content nodes in the graph.' This contradicts the abstract's phrasing that the solution adds interpretations 'based on the user activities' and the Section 3 statement that all experiments use CERT. The LLM demo should either be clearly labeled as a synthetic illustration, or the authors should show how CERT content is preprocessed and interpreted; in either case, the 80%/70% risk figures in Figure 8 need a provenance or a caveat.","section":"§5.3.2, Figure 8"}],"minor_comments":[{"comment":"There is an inconsistency between the general weighted logistic regression introduced at the start of §4.3, which fixes w_i = α for positive samples and w_i = 1 for negative samples, and the per-sample learned λ_i in Equation (2) and §5.2.3. Please reconcile these and state clearly which weighting scheme is actually used for the experiments.","section":"§4.3 and §4.3.1"},{"comment":"The stated uniform deviation bound O(√(log(1/δ)/n)) is too strong as written; for a bounded and Lipschitz loss, uniform convergence over a parameter set typically incurs a dependence on the dimension or covering number. The consistency claim is standard, but the proof sketch should be corrected to avoid an unrealistic rate.","section":"§4.3.1, proof of Theorem 1"},{"comment":"The term 'Gain' in Table 1 is not defined in the caption or text. Clarify whether it is recall at a fixed ranked cutoff and how the cutoff is chosen.","section":"§5.2.3, Table 1"},{"comment":"The phrase 'the percentage of threats detected is 0.34%' should be revised to 'the proportion of positive (threat) samples in the training data is 0.34%,' since detection rate is not a property of the data alone.","section":"§5.2.3"},{"comment":"The bullet stating that the LLM interprets risk 'based on its own training knowledge base from the whole Web' is vague and raises the risk of unsupported or hallucinated explanations. If this is a design choice, it should be justified and ideally evaluated by human reviewers; the paper currently provides no evaluation of explanation quality.","section":"§5.3.2"},{"comment":"The statement at the end of Section 3 that 'All experiments in this paper use this dataset' is contradicted by the use of online public documents for the LLM demo in Section 5.3.2. Please correct the inconsistency.","section":"§3 and §5.3.2"},{"comment":"Minor typos and capitalization issues appear, e.g., 'The Content of the user's Email, File, and Web Visits are important behavior factors.' Please proofread throughout.","section":"§5.3.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is clearly work in progress (it originated from a hackathon project), and the empirical section is not yet at the level needed for a journal publication. The most serious problem is that Equation (2), the centerpiece of the claimed improvement, is degenerate as printed; this must be fixed by specifying the actual constrained or penalized objective. If the authors can supply the correct objective, a cross-validated evaluation, and a clearer separation between the CERT-based quantitative results and the substituted-document LLM demo, the paper could be a reasonable preliminary/short contribution. I would also note that the key learnable-weight method is cited to the first author's prior work; the contribution relative to that work should be made explicit so that reviewers and readers can assess novelty fairly."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the system integration is real, and the demo is a useful artifact, but the paper's only quantitative result—the 3–4% gain—rests on a learning objective that is degenerate as printed. I wouldn't trust Table 1 until Eq. 2 is corrected and the experiments are redone.\n\nThe genuinely new piece is the combination: a knowledge graph that gets pruned and weighted by an imbalanced-learning model, with LLM agents handling retrieval and interpretation. That exact mix is not in the cited insider-threat work, and the architecture is clearly described. The authors are also honest that this is preliminary; Section 6 lists baselines, cross-validation, and human evaluation as future work, which is the right instinct.\n\nNow the soft spots. Eq. 2 says minimize over β and λ with no constraint on λ. For a positive sample (y_i=1), the term is −λ_i log π(x_i), and since log π < 0, the loss goes to −∞ as λ_i → −∞, or to 0 if λ_i is forced non-negative. So the printed objective never produces positive upweighting. Theorem 1 doesn't rescue it—it analyzes a fixed class weight α>1, not learned per-sample weights. The self-cited Zhang et al. (2022) may contain the real procedure, but it isn't reproduced, so the method behind Table 1 is underspecified. That is the load-bearing problem. Beyond that, the evaluation is one 70/30 split with no error bars, no baselines besides “with vs without weights,” no code, and the LLM interpretation is a demo where the CERT text had to be replaced with web documents because the real content was gibberish. The $200–400M loss-prevention figures are speculative arithmetic on top of a shaky result.\n\nWho is this for? A security practitioner looking for a system sketch of how KG, imbalanced learning, and LLMs could cooperate might get something out of Sections 4–5. But no one should cite the 3–4% number as evidence. My recommendation: send to peer review, because the integration is novel and the flaws are fixable in revision—but referees should demand a corrected learning objective, a proper experimental protocol, and a reproducible training procedure before any conclusions about the gain.","headline":"The system integration is real, but the central 3–4% gain is built on a learning objective that is degenerate as printed, so the empirical claim needs a corrected method and a proper evaluation before anyone should rely on it.","tokens_in":11958,"tokens_out":5028,"would_cite":false,"duration_ms":44086,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A knowledge graph, an imbalanced-learning classifier with learned per-sample weights, and an LLM interpreter together raise threat capture by 3–4% and add plain-English risk explanations.","keywords":["network threat detection","knowledge graph","large language model","imbalanced learning","multi-agent AI","weighted log-likelihood","insider threat detection","graph anomaly detection"],"falsifier":"Re-run the Model 1 versus Model 2 comparison under repeated k-fold cross-validation on the same dataset; if the top-3% capture-rate advantage does not persist across folds (or falls within noise), the claimed 3–4% gain is split-dependent. A complementary check is a label-shuffling test: if shuffled labels still produce a similar gain, the learned weights are absorbing noise rather than threat signal.","tokens_in":10869,"feed_emoji":"🛡️","tokens_out":14889,"duration_ms":114746,"temperature":0.7,"pith_summary":"The paper aims to establish that a three-part system detects insider threats better than a single classifier: a knowledge graph records user activities and their relationships, an imbalanced-learning model prunes weak or redundant graph information and scores the likelihood that a logon session is a known threat, and a large language model translates analyst questions into graph queries and writes plain-English explanations of the risk scores. On a public insider-threat dataset in which only 0.34% of logons are labeled threats, the authors show that a gradient-boosting model trained with per-sample weights learned from a custom log-likelihood captures more true threats among the riskiest logons than the same model trained without those weights: 60% versus 56% at the top 3% of predicted risky logons, and 98% versus 95% at the top 30%. The paper also proves that the weighted estimator is consistent with the usual $O(1/\\sqrt{n})$ generalization rate, bounds how far the weighted Jaccard graph-similarity score can drop under bounded activity changes, and frames the three agents' cooperation as a convex cooperative game with a non-empty core. If the central claim is right, the framework gives security analysts a modest but concrete gain in catching rare threats plus natural-language interpretations that connect each risk score to specific user activities.","feed_headline":"Weighted-log-likelihood training lifts threat capture by 3-4%","feed_subtitle":"A weighted classifier with knowledge-graph scoring and LLM explanations catches more real threats in the riskier logons.","key_machinery":"The load-bearing machinery is the weighted log-likelihood objective in Equation 2, in which each training sample $i$ receives its own learned weight $\\lambda_i$; these weights are learned from 70% of the data, then applied when training the gradient-boosting classifier, and the same weights weigh knowledge-graph nodes and edges so that strong threat signals are retained and weak or redundant signals are pruned. The second piece is the weighted Jaccard similarity $J(G,H)$ between a user's current and previous activity graphs, with the paper proving that a bounded total weight change $\\Delta$ relative to total weight $W$ keeps $J$ above $(W-\\Delta)/(W+\\Delta)$, so an abrupt drop in $J$ acts as a calibrated anomaly signal. The third piece is the LLM agent layer, which converts natural-language questions into graph queries, retrieves indexed text content, and generates human-readable risk interpretations; the agents' cooperation is modeled as a supermodular cooperative game, one in which adding an agent is more valuable when the other agents are already cooperating, so the Shapley value lies in the core.","core_discovery":"The central claim is that the three components work together rather than separately: the knowledge graph supplies structural activity patterns, the imbalanced-learning layer prunes and weights the graph and scores known threats, and the LLM retrieves and interprets the resulting evidence. The quantitative core is the learned-weight classifier: Model 2, a gradient-boosting model trained with per-sample weights $\\lambda_i$ from the custom weighted log-likelihood (Equation 2), outperforms Model 1 without those weights on the same validation data, capturing 60% versus 56% of true threats among the top 3% predicted risky logons and 98% versus 95% among the top 30%, with area under the precision-recall curve rising from 0.186 to 0.204. The paper further claims that a drop in weighted Jaccard similarity between a user's current and previous activity graphs flags unknown-threat behavior shifts, and that the LLM layer can summarize user activity and explain both known-threat and unknown-threat likelihoods in natural language, as demonstrated in an application demo.","pith_inferences":["Beyond the paper: the same learned-weight recipe is a general rare-event classifier, so it should be testable on other imbalanced security logs such as intrusion alerts or phishing reports, where the minority class is also below 1%.","Beyond the paper: the paper never measures whether the LLM's explanations are accurate or helpful; a human-subject study comparing analyst decision time and correctness with and without the explanations would settle that.","Beyond the paper: the Jaccard-drop anomaly signal and the weighted classifier are assumed complementary but never jointly evaluated; an injected-anomaly experiment could measure whether the similarity signal fires on novel attack patterns before the classifier has seen them.","Beyond the paper: the supermodularity of the cooperative game is assumed rather than estimated; computing empirical Shapley values from ablations of each agent on realistic workloads would test whether the core is actually non-empty for this system."],"forward_implications":["Security teams can review fewer logons to find the same threats: the learned-weight model catches 60% of true threats in the top 3% of predicted risky logons, versus 56% without weights, and 98% versus 95% in the top 30%.","The weighted Jaccard drop gives an independent signal for unknown threats, because an anomalous burst of new or changed activity edges pushes the graph-similarity score below its normal near-1 range.","Analysts get an on-demand plain-language explanation of why a user is risky, because the LLM retrieves the relevant subgraph and classifier scores instead of being fine-tuned on threat text.","The framework supports agile, online detection: new events are added as edges to the knowledge graph at each time step, so the graph and the anomaly score update as activity streams in.","The cooperative-game result guarantees that the three agents can be credited in a stable way, so no subset of agents has an incentive to break away from the collaboration."],"supporting_citations":[{"why":"Supplies the custom penalized log-likelihood from which the per-sample weights are learned; Model 2's 3–4% gain rests on this weighting scheme.","marker":"[Zhang et al., 2022]"},{"why":"Provides the public insider-threat dataset used for all experiments, including the 0.34% threat prevalence and the validation data behind the numbers in Table 1.","marker":"[Lindauer, 2020]"},{"why":"Supplies the weighted-likelihood consistency and asymptotic-normality results that the paper's Theorem 1 draws on.","marker":"[Xue et al., 2021]"},{"why":"Provides the convex-game result that the Shapley value lies in the core, used by Theorem 3 to show stable collaboration among the three agents.","marker":"[Shapley, 1971]"}],"fun_headline_variants":["Knowledge graph + LLM + imbalanced learning improves threat capture by 3-4%","Combining KG, LLM, and reweighting catches 3-4% more threats","Graph + LLM + imbalance-aware training boosts threat detection 3-4%","Threat capture up 3-4%: how KG, LLM, and reweighting work together","Weighted log-likelihood with KG and LLM lifts threat capture 3-4%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the per-sample weights learned from the custom log-likelihood on 70% of the historical data generalize to the held-out 30%, so the 3–4% capture-rate improvement reflects real signal rather than overfitting or a favorable split.","fun_headline_variants_meta":{"raw":{"variants":["Knowledge graph + LLM + imbalanced learning improves threat capture by 3-4%","Combining KG, LLM, and reweighting catches 3-4% more threats","Graph + LLM + imbalance-aware training boosts threat detection 3-4%","Threat capture up 3-4%: how KG, LLM, and reweighting work together","Weighted log-likelihood with KG and LLM lifts threat capture 3-4%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000866,"raw_usage":{"total_tokens":3723,"prompt_tokens":881,"completion_tokens":2842,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":497,"completion_tokens_details":{"reasoning_tokens":2722}},"tokens_in":497,"tokens_out":2842,"duration_ms":17279,"temperature":1.0,"reasoning_tokens":2722,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T14:13:09.311736+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the Model 1 versus Model 2 comparison under repeated k-fold cross-validation on the same dataset; if the top-3% capture-rate advantage does not persist across folds (or falls within noise), the claimed 3–4% gain is split-dependent. A complementary check is a label-shuffling test: if shuffled labels still produce a similar gain, the learned weights are absorbing noise rather than threat signal.","supporting_citations":[],"review_version":1}