{"id":"23a64fca-b60b-4708-a755-6dfd9ef74665","arxiv_id":"2607.13965","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"ProfMalPlus detects malicious NPM packages by extracting security-relevant code slices from object-sensitive behavior graphs and having coordinated LLM agents judge, enrich, and localize the evidence.","lead":"ProfMalPlus combines object-sensitive behavior graphs with a coordinated multi-agent LLM pipeline to detect malicious NPM packages, reporting a 98.1% F1-score on a curated benchmark and 597 newly found packages removed from NPM. It matters because supply-chain attacks through transitive dependencies are a growing, concrete threat, and the tool adds evidence-driven, interpretable detection for security teams.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Benchmark F1 is not representative: the curated set excludes obfuscated non-executable malicious packages (Sec 4.1), and the uncurated RQ5 results show precision 83.5% and F1 ≈ 87%, not 98.1%.","rationale":"The reader correctly identified the curated benchmark as the weakest assumption. I focused on the curation side because the paper explicitly acknowledges it (Sec. 5.1) and because it is testable. The RQ5 deployment is the best available check and already suggests a large gap between benchmark and uncurated performance: precision 83.5% and F1 ≈ 87% using the paper's own definitions. The memorization concern is real but secondary: RQ5's temporal separation addresses it for the real-world numbers, and if the curation test shows large recall loss on excluded packages, the benchmark claim is compromised independently of leakage. The proposed test uses existing ground truth (source datasets) and requires only rerunning the pipeline on the excluded set; it is feasible because the exclusion criteria are stated. This does not change the verdict: the paper is a solid systems contribution with a strong operational result (597 confirmed removals), but the headline F1 needs qualification.","tokens_in":43497,"tokens_out":7929,"duration_ms":79006,"concrete_test":"Run ProfMalPlus on the 568 malicious packages removed during curation (or the subset excluded for obfuscation/non-execution, with labels from the source datasets), using static analysis plus LLM reasoning with dynamic augmentation disabled; and on a random sample of benign packages with empty behavior graphs. Compute recall on the excluded malicious subset and F1 on the combined set. If recall on excluded malicious packages is substantially below 98.2% or combined F1 falls below 95%, the curation bias is real and the headline F1 should be reported with that caveat.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The 98.1% F1 in Table 2 is computed on the curated benchmark described in Sec. 4.1, not on an uncurated sample of npm. The curation is explicitly outcome-dependent: 13,467 malicious candidates were reduced to 1,658 by deduplication and then to 1,090 after manual filtering that excluded (among others) 'obfuscated packages that fail to execute in our Linux sandbox' and packages whose malicious behavior is not triggered at install/import time. Sec. 5.1 admits 'These removed packages might exhibit different malicious behaviors compared to the retained set.' On the benign side, only packages with at least one suspicious anchor were kept, so easy non-anchor benign packages are absent. This selection removes exactly the cases where the dynamic augmentor cannot help and where static analysis is hardest, so the resulting F1 is an upper-bound estimate for a filtered subpopulation, not a measure of real-world detection. The only leakage-free evaluation, RQ5 (Sec. 4.6), reports 597 true positives out of 715 alarms (precision 83.5%) and, using the paper's union-based false-negative definition, recall 90.7%, giving F1 ≈ 86.9%; because the union ground truth omits packages missed by all detectors, true recall/F1 may be even lower. Thus the headline 98.1% is not supported as a claim about general npm packages.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents ProfMalPlus, a detector for malicious NPM packages that combines object-sensitive behavior graphs with a coordinated multi-agent LLM reasoning pipeline. The pipeline constructs behavior graphs from entry files, extracts security-relevant code slices, runs local judge agents with self-consistency verification, then a global judge agent. For undetermined cases, a router selects either third-party enrichment (registry-derived module/method semantics) or dynamic augmentation (sandbox execution with instrumentation), feeding evidence back into slices. A localization agent finally reports malicious snippets. The evaluation claims a 98.1% F1-score on a curated benchmark of 1,090 malicious and 3,000 benign packages, outperforming five baselines, and reports a three-month deployment that detected 597 previously unknown malicious packages that were confirmed and removed from NPM.","tokens_in":43837,"tokens_out":3478,"duration_ms":39727,"significance":"If the reported effectiveness held on uncurated data, the paper would be a significant contribution to software supply-chain security. The system design is detailed and reproducible: the artifact release, prompt snippets, and component-wise ablation give the reader a concrete picture of how each stage contributes. The real-world deployment with 597 confirmed removals is a meaningful field result, and the interpretability evaluation (line-level localization F1 88.9% with inter-expert agreement reported) is a strength. The use of a temporal separation argument for RQ5 is also a step in the right direction for LLM-based detection. However, the evaluation protocol has several load-bearing weaknesses that currently prevent the headline numbers from being accepted as a statement about general NPM packages.","major_comments":[{"comment":"The benchmark underlying the headline 98.1% F1 is curated in an outcome-dependent way. Malicious packages were filtered to exclude those with no actual malicious code, non-executable obfuscated packages, and packages whose behavior is not triggered at install/import time; benign packages were retained only if they had at least one suspicious anchor. As §5.1 admits, the removed obfuscated packages \"might exhibit different malicious behaviors compared to the retained set.\" This selection removes exactly the cases where static/dynamic synergy cannot help and where static analysis is hardest, so the 98.1% is an upper-bound for a filtered subpopulation. The paper should report performance on the full unfiltered set or explicitly reframe the claim as applying only to this curated population.","section":"§4.1, Table 2"},{"comment":"The pipeline is stochastic (LLM-based), yet all reported point estimates (98.1% F1, ablation deltas) come from a single run with no confidence intervals, repeated-seed analysis, or variance reporting. The paper itself notes in §5.1 that LLM outputs are non-deterministic and that self-consistency \"cannot fully eliminate\" variance. The 3.5% gap over SocketAI and the 0.3% drop when removing third-party enrichment (Table 5) cannot be interpreted without knowing run-to-run variability. The authors should run the pipeline multiple times (or over multiple dataset folds) and report means, standard deviations, and CIs for the headline and ablation numbers.","section":"§4.2, Table 2; §5.1"},{"comment":"The only uncurated evaluation, RQ5, reports 597 confirmed true positives out of 715 alarms (precision 83.5%) and, using the union-of-detectors definition for false negatives, recall 90.7%, corresponding to F1 ≈ 87% — not 98.1%. The abstract and conclusion state 98.1% without noting that the RQ5 result is substantially lower. Moreover, §5.1 acknowledges that NPM does not disclose a complete malicious list, so the union-based ground truth may underestimate false negatives, making even the 87% F1 optimistic. The paper should present the RQ5 precision/recall/F1 as the primary real-world measure and reconcile the discrepancy with the curated-benchmark headline.","section":"§4.6, Table 7"},{"comment":"The RQ1 dataset is drawn from public malicious-package corpora (Backstabber's Knife, MalwareBench, OSCAR, MalOSS) that may overlap with the training data of the DeepSeek-V4-Flash backbone. The temporal-separation argument in §4.6 applies only to RQ5, not to the central RQ1 comparison, so memorization could inflate the 98.1% point estimate. The paper should provide a concrete leakage test — for example, comparing performance on public versus post-training-cutoff packages, or checking whether the LLM identifies known samples without evidence — or explicitly bound the effect of possible memorization on the RQ1 results.","section":"§4.1, §4.6"}],"minor_comments":[{"comment":"The ablated version name \"w/o MSC-Detection\" uses an acronym that is not expanded; for clarity it should be \"w/o Malicious Shell-Command Detection\" to match the description in §3.1.","section":"§4.4, Table 5"},{"comment":"Table 3's caption says \"Stage-Wise Decision Flow\" but the rows are detection runs (entry flows), not package-level totals. The caption should state this explicitly to avoid misinterpretation of the 89.9%/89.5% static-stage percentages.","section":"§4.2, Table 3"},{"comment":"Table 7's column headers \"FP Rate\" and \"FN Rate\" are not formally defined. Clarify the denominators (e.g., FP / total alarms, FN / (TP+FN)) either in the caption or the text.","section":"§4.6, Table 7"},{"comment":"The manuscript contains frequent missing spaces between words (e.g., \"proposeProfMalPlus\", \"ProfMalPlusachieves\", \"ProfMalPlusto\"). These appear to be formatting artifacts from PDF extraction but should be cleaned in the final version.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid systems contribution with a promising architecture and a valuable real-world deployment, but the evaluation protocol needs substantial strengthening before the headline claims can be accepted. The main risks are (1) the curated benchmark selection inflating RQ1 effectiveness, (2) absence of uncertainty quantification for a stochastic LLM pipeline, and (3) potential training-data leakage on public benchmark sets. I would request the additional experiments described in the major comments rather than rejecting: the method's core design and the RQ5 results suggest the approach has real value, and the issues are fixable within the scope of a revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Take the 597-package field result seriously; that's the real contribution. Take the 98.1% F1 with a grain of salt — it's measured on a curated benchmark that excludes hard cases (obfuscated packages that don't run in the Linux sandbox, packages with no suspicious anchor), and the paper's own RQ5 numbers on uncurated packages are more like 83.5% precision / ~87% F1. The abstract doesn't caveat this, though Section 5.1 does.\n\nWhat's genuinely new: the coordinated multi-agent loop — local judges with self-consistency, a router that picks between cheap third-party enrichment and sandbox dynamic augmentation, and a localization agent that spits out verbatim snippets. The ablation is honest: each component matters, and they admit the third-party enrichment's F1 gain is only 0.3% and mostly saves cost. The three-month deployment, with 597 confirmed removals and the lowest false-positive rate among the compared tools, is meaningful operational evidence. They also report cost and runtime, which is rare.\n\nSoft spots in proportion. The benchmark curation is the big one. Excluding non-executable obfuscated packages and keeping only benign packages with at least one suspicious anchor removes exactly the cases where static analysis and dynamic analysis are hardest, so the headline number is an upper bound for a filtered population, not a general claim. The RQ5 false-negative rate uses the union of detector alarms as ground truth, which can miss packages all detectors missed. The LLM pipeline is stochastic, and there are no confidence intervals or repeated-seed results. And the public malicious datasets used in RQ1 could be memorized by DeepSeek-V4-Flash; the temporal-separation argument applies only to RQ5. None of these kill the paper — the authors disclose most of them — but they mean the 98.1% should not be quoted without the caveats.\n\nWho it's for: people building or evaluating supply-chain detectors, and anyone interested in LLM-based program analysis with a realistic field deployment. It deserves a serious referee; the method is clearly specified, the evaluation is more transparent than most, and the field result is worth engaging with. My recommendation: send it to review, and make the reviewers push for variance analysis, an analysis of the excluded packages, and a better-defined real-world recall.","headline":"The 597-package field result is the real story; the 98.1% F1 is a curated-benchmark number, not a wild estimate.","tokens_in":44343,"tokens_out":2790,"would_cite":true,"duration_ms":30538,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ProfMalPlus detects malicious npm packages by feeding LLM agents annotated code slices from object-sensitive behavior graphs, reaching a 98.1% F1-score.","keywords":["malicious npm packages","supply-chain attack","LLM agents","behavior graph","code slicing","static analysis","dynamic analysis","malware localization"],"falsifier":"Run the same pipeline on npm packages published after the LLM's training cutoff, using independently confirmed takedowns as ground truth (as RQ5 does on a small scale): if the F1-score drops toward the 94.8% of the strongest baseline or below, the benchmark advantage is partly memorization. A second check: apply the detector to an unfiltered random sample of npm packages, including packages with empty behavior graphs, and see whether real-world precision remains near 83.5%.","tokens_in":43325,"feed_emoji":"📦","tokens_out":3337,"duration_ms":32054,"temperature":0.7,"pith_summary":"The paper claims that combining object-sensitive behavior graphs with coordinated LLM reasoning over annotated code slices detects malicious npm packages more accurately than existing detectors. Instead of classifying abstract graph features, it reconstructs security-relevant code with inline static, third-party, and runtime evidence, then lets judge agents reason over the code. The claim is supported by a 98.1% F1-score against five state-of-the-art detectors, a three-month deployment that found 597 previously unknown malicious packages, and line-level localization of malicious code. A sympathetic reader would care because it proposes a path from black-box detection to interpretable, evidence-driven auditing at registry scale.","feed_headline":"LLM agents plus behavior graphs catch npm malware at 98.1% F1","feed_subtitle":"It reviews annotated code slices through coordinated judge agents, then confirmed 597 new malicious packages in the wild.","key_machinery":"The central mechanism is the behavior graph: a directed graph over statements with control-flow, control-dependency, and data-dependency edges, built by object-sensitive traversal that tracks alias relations and qualified object paths so sensitive API calls (e.g., os.hostname hidden behind function aliases or obfuscated property access) can be resolved. From suspicious anchor nodes the code slicer extracts annotated source slices; local judge agents with self-consistency verification then reason over each slice, a global judge synthesizes entry-level verdicts, and a router selects third-party enrichment (registry-derived API semantics) or dynamic augmentation (sandbox execution resolving run","core_discovery":"On its own terms, ProfMalPlus claims that four limitations of prior npm malware detectors—poor modeling of obfuscated code, ignoring JavaScript's object-centric features, weak static-dynamic synergy, and semantic loss from abstraction—can be addressed in one pipeline. The pipeline builds a behavior graph with object-sensitive alias and property tracking, slides backward and forward from suspicious nodes to extract source-level code slices, annotates them with evidence, and runs a loop of local judge agents, a global judge, a router, and optional third-party or dynamic enrichment until a verdict is reached. The result is an entry-level malicious/benign decision with concrete code localization","pith_inferences":["I infer the benchmark advantage over the strongest baseline may be partly attributable to the LLM having seen historical malicious packages in training; the RQ5 deployment is the cleaner test, and its lower false-positive rate (16.5%) is the more convincing evidence.","I infer the pipeline's conservative policy, while reducing false positives, will continue to miss malware that only activates at explicit user invocation or under Windows-specific conditions; extending monitoring beyond install and import time is a natural next step.","I infer that combining this source-level reasoning with system-call-level monitoring could resolve the binary-execution false positives the paper identifies, since opaque binaries are currently flagged by default.","I infer the approach transfers to other package ecosystems (PyPI, RubyGems) that share install-time and import-time attack surfaces, though object-sensitive analysis would need adapting to each language's semantics."],"forward_implications":["If the claims hold, malicious npm packages can be detected at a 98.1% F1-score with concrete code localization, making audit results actionable rather than a bare label.","The static-first design (about 90% of verdicts reached without sandbox execution) means registry-scale screening can reserve expensive dynamic analysis for genuinely ambiguous cases.","Third-party enrichment resolves most routed third-party calls at about one-sixth the runtime of sandbox execution, so documentation-driven semantics can substitute for execution.","The real-world result of 597 confirmed removals suggests the approach generalizes beyond benchmark samples to newly published packages postdating the model's training cutoff.","Ablations show self-consistency verification and dynamic augmentation are the largest contributors; removing them drops F1 by 4.49 and 3.87 points respectively."],"fun_headline_variants":["LLM agents + behavior graphs spot npm malware at 98.1% F1","597 new npm malware sniffed out by LLM-coordinated graphs","ProfMalPlus agent graphs uncover 597 new malicious NPM packages","Static-dynamic graphs + LLM judges flush out 597 npm malware","LLM graph agents catch 597 npm malware, 98.1% F1"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The headline numbers rest on the assumption that the curated benchmark—1,090 malicious and 3,000 benign packages, filtered to those that execute in a Linux sandbox and expose at least one suspicious anchor—fairly represents real npm packages, and that LLM verdicts on these public historical samples are not inflated by training-data memorization.","fun_headline_variants_meta":{"raw":{"variants":["LLM agents + behavior graphs spot npm malware at 98.1% F1","597 new npm malware sniffed out by LLM-coordinated graphs","ProfMalPlus agent graphs uncover 597 new malicious NPM packages","Static-dynamic graphs + LLM judges flush out 597 npm malware","LLM graph agents catch 597 npm malware, 98.1% F1"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001405,"raw_usage":{"total_tokens":5530,"prompt_tokens":775,"completion_tokens":4755,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":519,"completion_tokens_details":{"reasoning_tokens":4655}},"tokens_in":519,"tokens_out":4755,"duration_ms":31830,"temperature":1.0,"reasoning_tokens":4655,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T03:09:45.900126+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same pipeline on npm packages published after the LLM's training cutoff, using independently confirmed takedowns as ground truth (as RQ5 does on a small scale): if the F1-score drops toward the 94.8% of the strongest baseline or below, the benchmark advantage is partly memorization. A second check: apply the detector to an unfiltered random sample of npm packages, including packages with empty behavior graphs, and see whether real-world precision remains near 83.5%.","supporting_citations":[],"review_version":1}