{"id":"aff414df-68e2-4e8c-a248-a19337489ef2","arxiv_id":"2505.03796","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":2.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"A vendor paper claims an AI-based insider risk scoring system cuts false positives by 59% and boosts true positives by 30%, but provides no reproducible evidence.","lead":"This paper describes a commercial insider-risk detection system that combines rule-based scoring, an autoencoder, and human feedback, and claims large reductions in false positives and response times. It is a product description with no released code, data, or evaluation protocol, so the performance claims cannot be independently checked.","discovery_kind":"incremental","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Section 4's headline metrics appear to be derived from the same analyst-feedback instances used to retrain the model; with no held-out evaluation described, the 59% FP reduction and 30% TP gain may be fitting artifacts rather than detection performance.","rationale":"The reader's verdict is REJECT, and my analysis supports that verdict. The decisive weakness is not the choice of baseline or the lack of novelty; it is that the evaluation section appears to measure performance on the same feedback instances that drive retraining. Section 3.2.3 describes incremental retraining on analyst feedback with feedback prioritized over initial weights, and Section 4.3 reports exactly the improvement trajectory of that training loop: FP falls 42% to 30% to 17%, and FN falls 18% to 15% to 12%. Section 4.1 states continuous evaluation on expert-annotated data but describes no held-out split, temporal holdout, or disjoint cohort. The parenthetical internal-methodology claims in Section 3.1 are flagged by the authors themselves as non-public and non-citable, which further rules out independent audit. The result is that Table 10's headline numbers are not distinguishable from memorization of the training labels, so the central claim of superior detection is unverified. The concrete temporal-holdout test I propose would settle this: if the claimed metrics survive evaluation on never-trained examples, the core detection claim would gain support; if they regress to the PRISM baseline, the rejection is clearly justified. The reader's weakest_assumption identified the same evaluation-leakage risk, so my agreement is 'agree'. The verdict remains REJECT, hence 'UNCHANGED' is the appropriate recommendation.","tokens_in":11517,"tokens_out":4046,"duration_ms":41014,"concrete_test":"The decisive check is a strict temporal holdout evaluation. Obtain the exact evaluation protocol and re-run Table 10 as follows: use the first N weeks, including the roughly 300 feedback instances from the initial three weeks and all retraining batches, to train and incrementally retrain the autoencoder; freeze the model at week 12; then compute FP, TP, and FN only on feedback-annotated examples that were never used in any training or retraining step, with a temporal gap after the last retraining batch. If the held-out numbers reproduce the claimed 17% FP, 85% TP, and 12% FN, the concern is resolved. If the numbers regress toward the PRISM baseline values (42%, 65%, 18%), the reported gains are fitting artifacts. A necessary part of the check is reporting the overlap count between test instances and retraining feedback instances used in Sections 4.1 and 4.3.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the AI-based adaptive scorer outperforms the static PRISM baseline (Table 10: FP 42% to 17%, TP 65% to 85%, FN 18% to 12%) and that feedback-driven retraining produces this gain. For that claim to be valid, the FP/TP/FN metrics must be measured on examples whose labels were not used to train or retrain the model. The paper never establishes this, and the evidence points the other way. Section 3.2.3 says the autoencoder is initially trained on PRISM-derived data and then incrementally retrained using analyst feedback, with 'user feedback is prioritized over initial training weights' (Section 3.2.3, step 4). Section 4.3 then reports FP/FN after the first and second feedback loops (42 to 30 to 17%, and 18 to 15 to 12%), exactly the trajectory one would see if the loop is evaluated on its own training corrections. Section 4.1 adds that the model was 'continuously evaluated on a dataset annotated by field experts and the administrator' after 12 weeks of feedback training, with no held-out split, no temporal gap, and no disjoint user cohort. The unverifiable internal-methodology annotations in Section 3.1 ('internal methodology – may not require citation', 'custom process – not publicly citable', 'practical engineering – internal claim') reinforce that the dataset and annotation protocol are not available for independent audit. Without an independent test set, the reported 59% false-positive reduction and 30% true-positive gain are equally consistent with overfitting to the analyst-corrected scores, so the comparison against PRISM does not support the paper's central claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript describes an AI-driven insider risk management system that combines a PRISM rule-based scorer, an autoencoder-based AI risk scorer with human feedback, policy-based violation detection, and LLM-generated recommendations. The authors report large improvements over the PRISM baseline (false positive rate 42% to 17%, true positive rate 65% to 85%, false negative rate 18% to 12%) and operational gains including a 47% reduction in incident response time. The evaluation is based on the CERT dataset augmented with production data and expert annotations.","tokens_in":11944,"tokens_out":3415,"duration_ms":32292,"significance":"If the reported gains were supported by a sound evaluation, the system would be a valuable practical contribution to insider risk management. The paper's strengths include a transparent PRISM scoring equation, a clear description of the human-in-the-loop retraining loop, and preliminary scalability measurements. However, the central comparative claim rests on an evaluation that is circular and lacks any held-out test set, so the empirical contribution cannot be accepted as stated.","major_comments":[{"comment":"The autoencoder is described in Section 3.2.3 as 'initially trained using data from the PRISM framework,' which means PRISM scores serve as training labels. Section 4.1 then claims the AI model outperforms PRISM (Table 10). This is not an independent comparison: the AI model inherits PRISM's scoring behavior and the comparison measures how well the autoencoder reproduced its labels. The authors must evaluate both methods on an independent test set that was neither used for initial training nor for feedback retraining, and report the PRISM and AIRS results separately on that set.","section":"Section 3.2.3, Section 4.1, Table 10"},{"comment":"Table 12 reports false positives falling from 42% to 30% to 17% across successive feedback loops, while Section 4.1 says the model was 'continuously evaluated on a dataset annotated by field experts and the administrator.' The paper never specifies a train/test split, a temporal gap, or a disjoint user cohort for this evaluation. If the same analyst-feedback instances used for retraining are also used for evaluation, the reported improvements are fitting artifacts rather than detection performance. An explicit description of evaluation data partitioning and a confirmation that test labels were not used in any retraining step are required.","section":"Section 4.3, Table 12"},{"comment":"The ground-truth annotation is described with bracketed notes '[internal methodology – may not require citation]', '[custom process – not publicly citable]', and '[practical engineering – internal claim]'. These statements make the annotation protocol and the resulting labels unavailable for independent audit. There is no definition of what constitutes a true insider threat, no inter-annotator agreement measure, and no description of how production streams were labeled. Without a verifiable ground truth, the FP, TP, and FN metrics in Table 10 are not reproducible.","section":"Section 3.1"},{"comment":"The scalability and incident-response improvements (sub-300ms latency at 100,000 logs/sec and a 47% reduction in response time) are reported without any experimental setup details: no hardware configuration, no workload description, no sample size, and no statistical uncertainty. These claims should be labeled as anecdotal or supported by a proper benchmark with reproducible methodology.","section":"Section 4.5, Section 4.6, Tables 14 and 15"}],"minor_comments":[{"comment":"There is a typo in the introduction: 'A WS IAM' should be 'AWS IAM'.","section":"Introduction, Section 1"},{"comment":"The text says 'as shown in Table 1' but the performance results are presented in Table 14; the reference should be corrected.","section":"Section 4.5"},{"comment":"Reference [13] has inconsistent formatting with a stray 'Glasser, Joshua and Lindauer, Brian' string; the entry should follow the standard author-list format used elsewhere.","section":"References"},{"comment":"Table 11 reports 'Actual Positive Rate' over time, but the term is not defined; it is presumably the true positive rate and should be stated as such for consistency with Table 10.","section":"Section 4.2"}],"recommendation":"reject","confidential_remarks":"The paper has a strong commercial-product character (all authors are from Vectoredge) and the evaluation appears designed to showcase deployment rather than to provide independent scientific validation. The explicitly non-citable internal methodology notes in Section 3.1 are a particular concern for a venue expecting reproducible research, and the circular training/evaluation loop is a load-bearing flaw that cannot be remedied by a minor revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is a vendor system description from Vectoredge that combines a weighted threshold rule (PRISM), an autoencoder trained on PRISM scores and analyst feedback, and a DeepSeek LLM for recommendations. The engineering integration is straightforward, and the architecture is described clearly. But the central performance claims are not supported by any describable evaluation. The comparison against PRISM is circular: the autoencoder is trained on PRISM output and then retrained on the same analyst corrections that are later reported as improvements. Without a held-out test set, the 59% false-positive reduction and 30% true-positive gain are fitting artifacts, not detection performance.\n\nWhat is actually new: nothing. The paper cites autoencoder-based insider threat detection [20] and human-in-the-loop refinement [21] as prior work. PRISM is a weighted sum with manually chosen weights. The contribution is an integration of off-the-shelf pieces into a product. There is no new algorithm, no new theory, no release of code or data, and no independent validation.\n\nWhere it does well: the paper gives a complete picture of a real system architecture, including the PRISM scoring equation, the feedback loop with alpha blending, the policy tables, and the scalability tests. The performance numbers (10M events/day, sub-300ms query latency) are concrete, though only the vendor's word backs them. The incident response reduction is plausible as an operational claim but lacks a controlled comparison.\n\nThe soft spots are large. Section 3.2.3 says the model is initially trained using PRISM data and then incrementally retrained with analyst feedback \"prioritized over initial training weights.\" Section 4.3 reports FP/FN dropping after each feedback loop, and Section 4.1 says the model was \"continuously evaluated on a dataset annotated by field experts and the administrator.\" No holdout split, temporal gap, confusion matrix, or statistical tests appear. The annotations are bracketed as \"internal methodology – may not require citation\" and \"custom process – not publicly citable,\" which effectively rules out independent audit. The free parameters (PRISM weights, thresholds, alpha, retraining trigger count) are chosen without sensitivity analysis.\n\nThe paper may describe a working product, but as a research claim it does not meet a minimal bar of verifiability. My recommendation: desk reject. For a serious referee to engage, the authors would need to release PRISM and AI scores on CERT with proper train/test splits and evaluate the feedback loop on fresh analyst judgments that were not used in retraining.","headline":"Vendor system description with circular evaluation; the reported FP/TP gains are fitting artifacts of retraining on the same analyst feedback used to measure them.","tokens_in":12458,"tokens_out":2016,"would_cite":false,"duration_ms":19150,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An autoencoder risk score seeded by a static PRISM baseline and refined through analyst feedback cuts false positives from 42% to 17% and raises true-positive detection from 65% to 85%.","keywords":["insider threat detection","insider risk management","risk scoring","autoencoder","behavioral analytics","human-in-the-loop","anomaly detection","false positive reduction"],"falsifier":"Rerun the evaluation on a strict holdout that excludes every event whose analyst adjustment fed the retraining loops, including the 12% of scores analysts changed. If the false positive rate rises back toward 42% or the true positive gain disappears, the reported improvement is feedback-loop fitting rather than detection skill. A companion test: in a live deployment, compare the AI score's ranking against a second analyst's independent judgment on events the model has never been retrained on.","tokens_in":11349,"feed_emoji":"🛡️","tokens_out":7533,"duration_ms":65042,"temperature":0.7,"pith_summary":"The paper claims that switching insider-risk scoring from a fixed weighted formula to an autoencoder that learns from analyst corrections can cut false positives by 59% (from 42% to 17%) and raise true-positive detection by 30% (from 65% to 85%). The reason this matters is that rule-based systems drown security teams in false alerts; an adaptive score that improves with each review could make insider-threat monitoring practical. The same system also reports a 47% reduction in average incident response time, from 45 to 24 minutes, at a scale of 10 million log events per day with sub-300ms queries.","feed_headline":"Adaptive scoring cuts insider-threat false positives from 42% to 17%","feed_subtitle":"Autoencoder plus analyst feedback beats static PRISM scoring and halves incident response time.","key_machinery":"The load-bearing piece is the autoencoder risk scorer trained on PRISM-labeled activity, assisted by a weighted linear baseline. PRISM sums seven weighted factors — privilege, activity type, application context, IP reputation, business hours, device compliance, and cumulative activity — into a normalized 0–1 score. The AI model replaces the weights with a learned reconstruction baseline: each new event gets a risk score from its reconstruction error, and analysts can adjust any score through $S_{\\mathrm{final}} = S_{\\mathrm{AI}} + \\alpha(S_{\\mathrm{user}} - S_{\\mathrm{AI}})$. Once enough adjusted instances accumulate, the autoencoder is retrained, prioritizing analyst feedback over its initial PRISM-derived weights. That feedback loop is what converts a static heuristic into an organization-specific, continuously updating risk measure.","core_discovery":"The central claim is that a hybrid scoring pipeline — a static PRISM baseline that seeds an autoencoder, which then assigns risk by reconstruction error and is incrementally retrained on analyst feedback — detects insider threats better than PRISM alone. On the paper's reported numbers, the AI score cuts the false positive rate from 42% to 17%, raises true positive detection from 65% to 85%, and lowers the false negative rate from 18% to 12% over a 12-week feedback process. The same deployment is said to process up to 10 million log events per day with sub-300ms query latency and to reduce incident resolution time from 45 to 24 minutes. The authors present these results as evidence that continuous human-in-the-loop adaptation, rather than more elaborate static rules, is the scalable path to insider-risk detection.","pith_inferences":["If the central mechanism is correct, the same pattern — a simple explainable baseline, an autoencoder seeded on it, and analyst-correction retraining — should transfer to other alert-triaging domains such as fraud or cloud-cost anomaly detection.","The paper's reported 12% analyst-adjustment rate is itself testable: if the feedback loop genuinely works, corrections should converge over time, and inter-analyst agreement on high-risk events could be measured to test whether the label noise is low.","The latency figures suggest the practical bottleneck is log ingestion and indexing, not the autoencoder; this predicts that larger embedding or graph-based features could be added without breaking the sub-300ms budget, which would be a cheap follow-up experiment."],"forward_implications":["If the metric gains hold, security teams would review fewer than half as many false alerts while catching more genuine insider threats, reducing alert fatigue.","An organization adopting the feedback loop would get a risk score that drifts toward its own analysts' judgment without rewriting policy rules.","The reported throughput — 10 million log events daily with sub-300ms queries — implies the scoring pipeline can run in real time at enterprise scale.","Automated enforcement triggered by policy violations (privilege revocation, file-access restriction, device disconnection) would shrink the window between detection and response.","The context-aware LLM recommendations, if they work as described, give analysts a plain-language reason for each high score, supporting faster triage."],"supporting_citations":[{"why":"Supplies the synthetic insider-threat activity logs used for training and evaluation.","marker":"[13, 14]"},{"why":"Provides the deep-autoencoder insider detection approach that motivates the reconstruction-error risk score.","marker":"[20]"},{"why":"Supports the human-in-the-loop feedback and incremental retraining design.","marker":"[21]"},{"why":"Defines the context-aware file sensitivity classifier used to flag PII/PHI/PFI before risk scoring.","marker":"[17]"},{"why":"Provide the risk-assessment and human-behavior modeling background that PRISM's weighted scoring builds on.","marker":"[18, 19]"},{"why":"Grounds the use of context-aware AI model recommendations that explain high-risk events to analysts.","marker":"[24]"}],"fun_headline_variants":["Insider-threat AI slashes false positives by 59%","42% to 17%: AI scoring cuts insider false alarms","AI insider-risk model: 30% better true positives, 59% fewer false alarms","Adaptive IRM: autoencoder + feedback trims false positives 59%","Cuts incident response 47%: adaptive AI reshapes insider-risk detection"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported gains depend on expert-annotated risk labels being correct ground truth and on the final evaluation using events the model had not already seen through its own retraining feedback.","fun_headline_variants_meta":{"raw":{"variants":["Insider-threat AI slashes false positives by 59%","42% to 17%: AI scoring cuts insider false alarms","AI insider-risk model: 30% better true positives, 59% fewer false alarms","Adaptive IRM: autoencoder + feedback trims false positives 59%","Cuts incident response 47%: adaptive AI reshapes insider-risk detection"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001066,"raw_usage":{"total_tokens":4468,"prompt_tokens":942,"completion_tokens":3526,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":558,"completion_tokens_details":{"reasoning_tokens":3425}},"tokens_in":558,"tokens_out":3526,"duration_ms":24464,"temperature":1.0,"reasoning_tokens":3425,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:34:30.524810+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun the evaluation on a strict holdout that excludes every event whose analyst adjustment fed the retraining loops, including the 12% of scores analysts changed. If the false positive rate rises back toward 42% or the true positive gain disappears, the reported improvement is feedback-loop fitting rather than detection skill. A companion test: in a live deployment, compare the AI score's ranking against a second analyst's independent judgment on events the model has never been retrained on.","supporting_citations":[{"cited_title":"Insider Detection using Deep Autoencoder and Variational Autoencoder Neural Networks","cited_arxiv_id":"2109.02568","evidence_quote":"Provides the deep-autoencoder insider detection approach that motivates the reconstruction-error risk score."},{"cited_title":"Humans in AI: The necessity for human-in-the-loop (HILT).Se- curity Magazine, 2024","cited_arxiv_id":null,"evidence_quote":"Supports the human-in-the-loop feedback and incremental retraining design."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the context-aware file sensitivity classifier used to flag PII/PHI/PFI before risk scoring."},{"cited_title":"Elevating Threat Intelligence: Integrating Context-Aware AI Models for Real-Time Cyber Defense.Wrixte, May 15, 2024","cited_arxiv_id":null,"evidence_quote":"Grounds the use of context-aware AI model recommendations that explain high-risk events to analysts."}],"review_version":1}