{"id":"213e9ade-f37f-4b3f-8580-7999cd943371","arxiv_id":"2509.01509","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"Insight-LLM fuses four behavioral views via Qformer adapters and a LoRA-tuned LLM, reporting state-of-the-art insider threat detection on CERT r4.2/r5.2, with unresolved reporting inconsistencies.","lead":"A team applies a large-language-model pipeline to insider threat detection, fusing text, sentiment, activity sequences, and interaction graphs through adapters and a LoRA-tuned 8B LLM. The paper reports top scores on two CERT benchmarks, but the supporting numbers are inconsistent in places and the promised latency gains are never measured.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The F1 values in Table 2 are internally inconsistent with the reported Precision and DR under any standard F1 definition, so the headline SOTA claim is not reproducible from the paper's own numbers.","rationale":"The reader's weakest_assumption is that the evaluation protocol is unbiased and the reported numbers are accurate, and it explicitly flags that the F1 column does not reconcile with precision and DR. My independent computation confirms the internal inconsistency is concentrated in the very metric (F1) that drives the SOTA claim. The F1 numbers are not derivable from the table's own reported Precision and DR under the standard formula, and the paper gives no alternative definition. The correct verdict remains CONDITIONAL rather than REJECT: the architecture and ablations are coherent, and the inconsistency is a reporting/verifiability defect that code release or a corrected metric definition could resolve. I do not elevate to ACCEPT because the load-bearing number is currently unverifiable; I do not lower to REJECT because there is no evidence of fraud and the concern is fixable. I agree with the reader's emphasis: this is the first thing to check. The other issues (no seeds, user-overlapping split, missing latency/parameter counts, section heading mismatch) are secondary to the arithmetic inconsistency, and the abstract's low-latency claim cannot be assessed without the missing measurements, but the F1 discrepancy is the most concrete and load-bearing.","tokens_in":22196,"tokens_out":1669,"duration_ms":14972,"concrete_test":"Recompute every F1 cell in Table 2 from the table's own Precision and DR columns using F1=2PR/(P+R), and also attempt per-user and per-session aggregations if the raw prediction logs can be recovered from the authors. If no aggregation rule reproduces the reported F1 values, the headline SOTA claim is unsupported. Independently, run the released (or author-provided) code on CERT r4.2/r5.2 with the Section 4.4 configuration and the stated 2010-train/2011-test split to see whether Table 2's numbers are reproduced within a seed; report seeds and standard deviations.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is 'Insight-LLM achieves SOTA F1 0.9712/0.9594 on CERT r4.2/r5.2'. But Table 2's own three columns do not reconcile: for the headline row on r4.2, Prec=0.9631, DR=0.9683 gives harmonic-mean F1 = 2*0.9631*0.9683/(0.9631+0.9683) = 0.9657, not 0.9712. For r5.2, 2*0.9512*0.9466/(0.9512+0.9466) = 0.9489, not 0.9594. The same inconsistency appears in multiple baseline rows (e.g., LAN r4.2: 2*0.9258*0.9478/(0.9258+0.9478)=0.9367 vs reported 0.9492). No metric definition is given. If F1 is computed by some other aggregation (e.g., per-user averaging or sample-weighted averaging over sessions), it must be stated; if it is standard macro F1, the table is arithmetically wrong. Because the F1 improvement over LAN is the headline result (F1 0.9712 vs 0.9492; 0.9594 vs 0.9156), and the unreconciled numbers appear exactly in the comparison's numerator, the reported superiority is not internally established without code or clarification. This is a load-bearing internal-consistency issue, not a disagreement with consensus: the reader cannot verify the central number from the paper alone.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Insight-LLM, a multi-view fusion framework for insider threat detection. It defines four behavioral views (textual, sentiment, sequential, topological), encodes each with a frozen pretrained encoder (RoBERTa, sentiment RoBERTa, TS2Vec, Node2Vec), maps the features into an LLM's semantic space with per-view Qformer adapters, fuses them via cross-view attention, and classifies by cosine similarity between the last hidden state and verbalizer tokens after LoRA tuning of Llama-3-8B. Experiments are conducted on CERT r4.2 and r5.2 against six baselines; the paper reports state-of-the-art F1 of 0.9712/0.9594 and FPR of 0.0476/0.0496, along with ablations, hyperparameter sensitivity, and prompt/label-word studies.","tokens_in":22456,"tokens_out":10206,"duration_ms":102270,"significance":"If the reported results hold, Insight-LLM is a technically plausible and timely contribution: it combines frozen per-view encoders, Qformer-based alignment into an LLM semantic space, cross-view attention, and LoRA-tuned LLM classification in a modular way that avoids training separate large models per view. The ablation study (Table 3) is informative, and the direction of the component contributions aligns with the paper's narrative. The algorithm box and sensitivity analyses are also helpful. However, the paper currently lacks the supporting evidence needed to certify the central SOTA claim: the reported F1 values are internally inconsistent, the evaluation protocol may permit user-identity leakage, and the claimed low latency/parameter overhead is not measured. If these issues are resolved (e.g., with corrected metrics, a user-disjoint split, and efficiency measurements), the contribution would be significant for the insider-threat-detection community.","major_comments":[{"comment":"The reported F1 values do not match the harmonic mean of the same row's Precision and DR under the standard F1 definition. For r4.2, Insight-LLM: 2*0.9631*0.9683/(0.9631+0.9683)=0.9657, but Table 2 lists 0.9712. For LAN: 0.9367 vs 0.9492; for r5.2 Insight-LLM: 0.9489 vs 0.9594. The discrepancy appears in both directions in Table 3 as well (e.g., w/o Text r4.2: computed 0.6842 vs listed 0.6755). No metric definition is given anywhere. Since F1 is the headline SOTA metric in the Abstract and §5.1, the table is internally inconsistent and must be reconciled by stating the exact aggregation (per-user/session macro, sample-weighted, etc.) or correcting the numbers. This is load-bearing for the central claim.","section":"Table 2, §5.1"},{"comment":"The train/test split uses all 2010 data for training/validation and Jan-Jun 2011 for testing, with the same employees in both periods. Because CERT's threat scenarios unfold over months and inject abnormal activities per user, anomalous activities of the same malicious employees can appear in both training and test sets. The paper does not rule out this user-identity leakage, and no user-disjoint or label-disjoint evaluation is reported. The reported improvements over LAN may therefore reflect memorization of user scenarios rather than multi-view generalization. Please clarify whether any malicious user's abnormal activities appear in the training period and, if so, provide additional experiments with a user-disjoint split or equivalent safeguards. Also, all results are single-run with no seeds/error bars, so the significance of the FPR drops cannot be assessed.","section":"§4.3 Datasets and Evaluation Protocol"},{"comment":"The abstract claims 'low latency and parameter overhead', but the paper reports no latency, throughput, memory, or parameter counts. §5.4 is titled 'Parameter efficiency' but contains only F1/FPR sensitivity sweeps over fuse-token count, LoRA rank, and α; no efficiency metric is measured. Either provide actual measurements (e.g., inference time, trainable parameters) or remove/qualify the claim.","section":"Abstract; §5.4 Parameter efficiency"},{"comment":"The sensitivity and prompt/label-word experiments report F1 values (e.g., 0.9394) that are substantially lower than the full-model F1 of 0.9712/0.9594 in Table 2, and the dataset/split on which these numbers are computed is not stated. Are these on a validation subset? On CERT r4.2 only? Without this context, RQ4/RQ5 results cannot be related to the main evaluation, and the final model configuration is not clearly the configuration used in §5.4 (e.g., α=0.7 vs the LoRA α=16 of §4.4). Please specify the evaluation protocol for each analysis.","section":"§5.4 and §5.5"}],"minor_comments":[{"comment":"The prose refers to 'line 7', 'line 10', 'lines 12–13', etc., but the displayed algorithm has different line numbering (e.g., feature extraction is line 2, not line 7). Please synchronize the references.","section":"Algorithm 1 and §3.4"},{"comment":"The title 'Impact of Base LLMs' is misleading: the experiments vary prompt and verbalizer designs, not the base LLM. Rename the section accordingly.","section":"§5.5"},{"comment":"The caption mentions 'Parse-LLM' rather than 'Insight-LLM'.","section":"Figure 4 caption"},{"comment":"References [47] and [48] are for OpenLLaMA/LLaMA, not for Meta-Llama-3-8B as cited. Add the correct Llama 3 reference.","section":"§4.4"},{"comment":"The index terms list 'Chain of Thought' and 'Multi-Agent', but the paper does not use CoT or multi-agent reasoning. Adjust the keywords to match content.","section":"Index Terms"},{"comment":"The LoRA update is written as ∆W·XLLM on the whole input embedding. Clarify that LoRA adapts specific attention/MLP projection matrices and align the notation with Algorithm 1.","section":"Eq. (16)"},{"comment":"The authors note that the weight-allocation analysis 'requires further validation via per-sample analysis and ablation.' This caveat should be acknowledged in the conclusions or addressed with per-sample evidence.","section":"§5.3"},{"comment":"The pilot experiment behind Figure 1 lacks dataset/split details; add a sentence describing what data produced Figure 1.","section":"§1 Pilot Experiments"},{"comment":"The 'Improvements' row is not a baseline; consider moving the relative gains into the text or removing this row.","section":"Table 2"}],"recommendation":"major_revision","confidential_remarks":"The main blocker is the unreconciled F1 numbers: the headline SOTA comparison is not verifiable from the paper's own table without a precise metric definition or code. The user-overlap evaluation protocol is also a serious risk. If the authors provide code/data and address these points, I would be inclined to support acceptance; in the current state, the central empirical claim is not established."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the architecture is the most interesting part—frozen per-view encoders, Qformer adapters, cross-view attention reweighting, and a LoRA-tuned LLM with verbalizer classification is a sensible way to fuse heterogeneous behavior logs. The paper explains the three ITD-specific challenges (parameter growth, semantic misalignment, view imbalance) and the design maps onto them cleanly. Ablations behave as expected: removing text hurts most, sequence next, and the fusion modules each contribute. That part is solid.\n\nThe problem is the evidence. The headline F1 values in Table 2 do not reconcile with the same row's precision and DR under any standard F1 definition. For r4.2, the reported Prec 0.9631 and DR 0.9683 give a harmonic-mean F1 of about 0.9657, not 0.9712. The LAN baseline row gives 0.9367, not 0.9492. The same pattern appears on r5.2. No metric definition is given. If F1 is macro-averaged per user or computed some other way, the authors need to state it; if it's standard F1, the table is arithmetically wrong. Because the SOTA claim is built on that F1 delta, this is load-bearing, not cosmetic.\n\nThere are also protocol concerns. The split uses 2010 activities for training and Jan–Jun 2011 for testing, but the same employees appear in both periods, and CERT injects anomalies per user over time. That lets the model memorize user or scenario identity. No seeds, no error bars—every table is a single run. The abstract promises 'low latency and parameter overhead' but no latency or parameter counts appear anywhere. And Section 5.4 reports validation F1 0.9394, FPR 0.0296 for the default config, while Table 2 says test F1 0.9712, FPR 0.0476; the averaging basis and the relationship between these two sets are unstated. Minor but telling: Figure 4's caption says 'Parse-LLM', and Section 5.5's heading says 'Impact of Base LLMs' while the content is about prompts and label words.\n\nSo: the machinery is plausible and the write-up is honest about prior art, but the reported numbers are not self-consistent, and the evaluation protocol has a known leakage risk. I wouldn't cite this as a SOTA result yet. It does deserve a real referee—a good reviewer can separate the architecture from the broken table—but the revision bar should be high: fix the F1 definitions, report variance, and use a leak-proof split.\n\nRecommendation: send to peer review, with the expectation of major revision.","headline":"The architecture is a sensible multi-view fusion for ITD, but the headline F1s don't reconcile with the reported precision/recall and the split leaks user identity, so the SOTA claim isn't established from the paper alone.","tokens_in":23089,"tokens_out":3383,"would_cite":false,"duration_ms":33023,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Insight-LLM claims that fusing four behavioral views into an LLM-based pipeline detects insider threats with F1 0.9712/0.9594 and false-positive rates below 5% on the two benchmark datasets.","keywords":["insider threat detection","multi-view fusion","large language models","Qformer","cross-view attention","LoRA","behavioral anomaly detection","CERT datasets"],"falsifier":"Re-run the exact pipeline under a user-disjoint split—train on a set of employees whose activity appears only in 2010, test on different employees in 2011—with multiple random seeds and report F1 and FPR. If F1 falls to the LAN baseline level or FPR climbs above 5%, the central claim collapses. Also reconcile Table 2's F1 0.9712/0.9594 with the parameter-study text reporting F1 0.9394 on what appears to be the same configuration.","tokens_in":21892,"feed_emoji":"🛡️","tokens_out":8179,"duration_ms":85272,"temperature":0.7,"pith_summary":"Insider threat detection needs to catch rare, scattered signals across many kinds of user behavior. The paper argues that single-view detectors miss too much, while existing multi-view methods become expensive, semantically misaligned, and biased toward dominant views. Insight-LLM is proposed as the first multi-view fusion framework built for this problem: frozen pretrained encoders extract text, sentiment, sequence, and topology embeddings; dedicated Qformer-style adapters align them into a large language model's semantic space; cross-view attention re-weights weak threat cues; and a LoRA-tuned LLM with verbalizer tokens performs classification. On two standard insider-threat datasets, the paper reports state-of-the-art F1 and large reductions in false positives. If these results hold, the architecture offers a scalable template for fusing heterogeneous security logs without retraining large per-view models.","feed_headline":"LLM fusion lifts insider-threat F1 to 0.9712","feed_subtitle":"Four behavior views—text, sentiment, sequence, topology—cut false alarms by up to 61% on the two benchmarks.","key_machinery":"The central machinery is a four-stage pipeline: (1) frozen, unsupervised pretrained encoders produce view-specific embeddings; (2) per-view ViewAdapters—Qformer-style cross-attention modules with learnable query tokens—project each view into the LLM's semantic manifold; (3) an inter-view cross-attention layer with learnable fusion queries re-weights and combines the aligned views; (4) the fused tokens plus a natural-language prompt are fed through a LoRA-tuned LLM, and classification is read out as cosine similarity between the final hidden state and verbalizer tokens. This design lets the model leverage the LLM's pretrained semantic priors while keeping trainable parameters small.","core_discovery":"Insight-LLM claims that insider threat detection can be cast as multi-view fusion and solved with a modular architecture built around a frozen large language model. Four behavioral views—text, sentiment, sequence, and topology—are each encoded by a separate frozen pretrained encoder, then projected by a dedicated ViewAdapter (a Qformer-style cross-attention module with learnable query tokens) into the LLM's shared semantic manifold. An inter-view cross-attention layer re-weights the aligned views so weak but consistent threat cues are amplified and dominant or noisy views are down-weighted. The fused token sequence, concatenated with a natural-language prompt, goes to a LoRA-tuned LLM, and t","pith_inferences":["The training and test periods overlap in users: the same employees appear in the 2010 training data and the 2011 test data, so part of the reported gain may come from user-level memorization rather than generic anomaly detection; a user-disjoint split would quantify this.","The paper reports F1 0.9712 on CERT r4.2 in Table 2, while the parameter and prompt studies in Sections 5.4–5.5 report a best F1 of 0.9394 on what appears to be the same configuration; these numbers cannot both describe the same evaluation unless different splits or validation sets were used.","The ablation shows removing text collapses F1 to roughly 0.68 and removing sequence drops it to about 0.78, while removing topology or sentiment costs only a few points; the 'multi-view' advantage may therefore be primarily a text-plus-sequence fusion with modest complementary gains from the other views.","If validated on user-disjoint splits, the frozen-encoder-plus-adapter design could transfer to other sparse, heterogeneous log domains—cloud access audits, medical record access, or financial transaction logs—where labeled anomalies are rare and semantic alignment is needed to surface weak cross-modal cues."],"forward_implications":["Insider threat detection can be built as a modular pipeline: adding a new behavioral view requires only a frozen encoder and a small adapter, avoiding the parameter explosion of training separate per-view sub-models.","Semantic alignment into a shared LLM manifold allows weak views like sentiment and topology to contribute without being drowned out, because cross-view attention reweights them by contextual relevance.","Verbalizer-based readout means class descriptions outperform arbitrary or numeric labels, so the LLM's language priors directly improve detection under label sparsity.","If the reported numbers hold, deployment would see roughly 42–61% fewer false alarms than the strongest compared baseline at comparable or better detection rates, reducing alert fatigue in security operations.","Because the text and sequence views carry most of the signal, the framework's robustness depends on keeping those encoders high-quality while using the attention modules mainly to suppress noise from weaker views."],"supporting_citations":[{"why":"Supplies the Qformer cross-attention design that the per-view ViewAdapters use to align heterogeneous views into the LLM semantic space.","marker":"[14]"},{"why":"Supplies the frozen text encoder used for the textual view.","marker":"[9]"},{"why":"Supplies the frozen unsupervised time-series encoder used for the sequence view.","marker":"[10]"},{"why":"Supplies the graph embedding encoder used for the topology view.","marker":"[11]"},{"why":"Supplies the sentiment-adapted text encoder used for the sentiment view.","marker":"[34]"},{"why":"Defines the hour-bucketing of event types and timestamps and contributes the ITDBERT baseline.","marker":"[35]"},{"why":"Provides the LAAEB fusion baseline and motivates the parameter-scalability challenge of late fusion.","marker":"[3]"},{"why":"Provides the LAN baseline—the strongest comparison—and the session-splitting protocol used to construct activity samples.","marker":"[8]"},{"why":"Introduces LoRA, the parameter-efficient adaptation method used to tune the LLM with minimal overhead.","marker":"[37]"},{"why":"Supplies the two benchmark datasets that all experiments are run on.","marker":"[46]"}],"fun_headline_variants":["Insight-LLM fuses four views to catch insider threats","Frozen LLM plus multi-view fusion nails insider detection","Insider threats: multi-view LLM fusion slices false alarms","Four behavior views, one LLM: insider threat win"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The load-bearing premise is that the evaluation protocol is fair and the reported numbers are accurate: the same employees appear in the training and test periods, every result is a single run with no seeds or error bars, and the F1/FPR values in different sections do not reconcile; if any of these fail, the claimed advantage over baselines is not established.","fun_headline_variants_meta":{"raw":{"variants":["Insight-LLM fuses four views to catch insider threats","Frozen LLM plus multi-view fusion nails insider detection","Insider threats: multi-view LLM fusion slices false alarms","Four behavior views, one LLM: insider threat win"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000191,"raw_usage":{"total_tokens":1129,"prompt_tokens":645,"completion_tokens":484,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":389,"completion_tokens_details":{"reasoning_tokens":423}},"tokens_in":389,"tokens_out":484,"duration_ms":5713,"temperature":1.0,"reasoning_tokens":423,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T12:30:07.836694+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the exact pipeline under a user-disjoint split—train on a set of employees whose activity appears only in 2010, test on different employees in 2011—with multiple random seeds and report F1 and FPR. If F1 falls to the LAN baseline level or FPR climbs above 5%, the central claim collapses. Also reconcile Table 2's F1 0.9712/0.9594 with the parameter-study text reporting F1 0.9394 on what appears to be the same configuration.","supporting_citations":[{"cited_title":"BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models,","cited_arxiv_id":null,"evidence_quote":"Supplies the Qformer cross-attention design that the per-view ViewAdapters use to align heterogeneous views into the LLM semantic space."},{"cited_title":"Ts2vec: Towards universal represen- tation of time series,","cited_arxiv_id":null,"evidence_quote":"Supplies the frozen unsupervised time-series encoder used for the sequence view."},{"cited_title":"node2vec: Scalable feature learning for networks,","cited_arxiv_id":null,"evidence_quote":"Supplies the graph embedding encoder used for the topology view."},{"cited_title":"ITDBERT: temporal-semantic representation for insider threat detection,","cited_arxiv_id":null,"evidence_quote":"Defines the hour-bucketing of event types and timestamps and contributes the ITDBERT baseline."},{"cited_title":"Laaeb: A comprehensive log-text analysis based approach for insider threat detection,","cited_arxiv_id":null,"evidence_quote":"Provides the LAAEB fusion baseline and motivates the parameter-scalability challenge of late fusion."},{"cited_title":"LAN: learning adaptive neighbors for real-time insider threat detection,","cited_arxiv_id":null,"evidence_quote":"Provides the LAN baseline—the strongest comparison—and the session-splitting protocol used to construct activity samples."},{"cited_title":"Lora: Low-rank adaptation of large language models,","cited_arxiv_id":null,"evidence_quote":"Introduces LoRA, the parameter-efficient adaptation method used to tune the LLM with minimal overhead."},{"cited_title":"Few-shot insider threat detection,","cited_arxiv_id":null,"evidence_quote":"Supplies the two benchmark datasets that all experiments are run on."}],"review_version":1}