{"id":"4231dbad-c889-41f1-8f1e-9b48744ea9b2","arxiv_id":"2602.16304","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"LLMs detect malicious PyPI packages reasonably at package level but identify specific malicious code indicators poorly, a 'granularity gap' that limits them to triage.","lead":"This paper tests 13 large language models on spotting malicious Python packages, first at the package level and then at the level of individual suspicious lines of code. It finds models can flag bad packages fairly well but are much worse at saying exactly which code behavior makes a package malicious.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 41% granularity gap rests on a 10-malicious-package binary sample and configuration-specific 37-file subsets; the gap may be a sampling artifact.","rationale":"The central claim is a quantitative gap: near-perfect binary detection vs. 41% degradation in indicator identification. This claim requires both metrics to be measured on representative, comparable samples. Section 4.5 violates this: the binary task uses 2.5% of malicious packages (10 samples) and 2.5% of benign (93 samples), while the multi-label task uses a different 37-file subset per configuration. With only 10 positive samples, a single misclassification changes F1 by ~0.05; the 0.99 F1 for GPT-4.1 means it classified essentially all 10 correctly, which is fragile and potentially explained by memorization (Section 6 acknowledges contamination risk). The multi-label means are aggregated over different subsets, making cross-configuration averages uninterpretable. These issues are more load-bearing than ground-truth label correctness because even perfect labels cannot fix the statistical incomparability. The reader's rationale mentions the sampling problems but the formal weakest assumption is label accuracy; our concern is complementary. A matched, larger-scale rerun would settle whether the 'granularity gap' is a real phenomenon or an artifact of tiny, non-overlapping samples. Thus the verdict remains CONDITIONAL, not REJECT, because the qualitative direction (package-level easier than indicator-level) is plausible and supported by reasoning analysis.","tokens_in":27478,"tokens_out":3433,"duration_ms":30172,"concrete_test":"Recompute both tasks on matched, large samples: binary classification on all 370 malicious and all 3,700 benign packages (or a 50% stratified sample), and multi-label identification on a single fixed random 100-file subset used identically by every model/configuration. If GPT-4.1's binary F1 falls below ~0.90, or if the binary-to-multi-label gap differs from 41% by more than 10 percentage points, the granularity gap claim is not robust. Additionally, test contamination by evaluating on malicious packages published after GPT-4.1's knowledge cutoff.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.5 samples only 2.5% of the binary dataset: 10 malicious and 93 benign packages. GPT-4.1's binary F1 ≈ 0.99 and the aggregate binary mean (µ=0.58) are computed on these 10 malicious samples; with n=10 the 95% confidence interval for F1 is extremely wide, and Section 6 acknowledges that closed models may have memorized these specific packages. For Task 2, each configuration uses a different 37-file subset of the 396-file malicious dataset, so the multi-label means (µ=0.34) are averages over non-identical test sets. The headline '41% decline' subtracts two incomparable, undersampled numbers. Even if the ground-truth labels are perfect, the size of the granularity gap is not statistically established. The qualitative direction may hold, but the quantitative claim and the 'near-perfect' binary performance are not reliable.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper evaluates 13 LLMs on two tasks: binary detection of malicious PyPI packages and fine-grained identification of specific malicious indicators (lines of code). Using a dataset of 4,070 packages (370 malicious, 3,700 benign) and a taxonomy of 47 indicators, the authors report a \"granularity gap\": GPT-4.1 achieves near-perfect binary F1 (≈0.99), while performance drops by approximately 41% for indicator-level identification. They also analyze the influence of five prompting strategies, three temperature settings, parameter size, context width, and code complexity, and recommend using LLMs for initial triage rather than autonomous indicator attribution.","tokens_in":27715,"tokens_out":6439,"duration_ms":63327,"significance":"If the granularity gap is robust, the paper provides a practically useful benchmark: LLMs can serve as triage tools but are not yet reliable for precise, indicator-level attribution. The study is broad in coverage (13 models, multiple prompts, temperatures, and configurations) and the authors state that a replication artifact is available. However, the central quantitative claims currently rest on small, non-comparable samples and on author-generated ground-truth annotations that are not externally validated. The direction of the granularity gap is plausible, but the specific 41% figure and the \"near-perfect\" binary performance are not statistically established. The recommendation to use LLMs for triage is reasonable, but it should be decoupled from the precise numerical claim until the sampling and validation concerns are resolved.","major_comments":[{"comment":"The headline \"41% granularity gap\" is not statistically established. Task 1 binary F1 (including GPT-4.1's 0.99) is computed from 10 malicious and 93 benign packages; Task 2 means are averaged over configuration-specific 37-file subsets. The subtracted quantities (µ_b=0.58 vs µ_m=0.34) come from different test sets and sample sizes, and no confidence intervals or paired significance tests are reported. Section 6 acknowledges sampling limitations but does not quantify their effect. Please report bootstrap CIs, use a fixed test set for both tasks, or run on the full datasets before claiming a specific percentage decline.","section":"§4.5, §5.1.5"},{"comment":"The abstract and conclusion state that \"longer packages are harder to analyze\" and that code complexity has a meaningful impact, but §5.3.5 reports Spearman ρ=-0.04 (p<0.01) with \"negligible explanatory power.\" The same paragraph rejects the null hypothesis and then states complexity does not significantly affect rank. This is internally inconsistent and should be corrected; if the claim is about LOC rather than cyclomatic complexity, that analysis is not reported.","section":"§5.3.5, Abstract, §7"},{"comment":"All Task 2 scores and the granularity gap depend on the statement-level, 47-indicator ground truth from the authors' prior paper [9]. Section 6 states \"We rely on the accuracy of the ground truth annotations,\" but no inter-annotator agreement or external validation is provided. Because the central quantitative claim is computed against this annotation, its correctness is load-bearing. Please report annotation reliability or demonstrate robustness of the conclusions to label noise.","section":"§4.2, §6"},{"comment":"GPT-4.1's \"near-perfect\" binary F1 rests on only 10 malicious packages, and Section 6 acknowledges that closed models may have memorized the test packages. With such a small positive sample, F1=0.99 is fragile and the memorization risk is not controlled. Please provide contamination checks (e.g., training-data cutoff analysis, deduplication) or temper the opening claim to reflect this uncertainty.","section":"Table 2, §6"}],"minor_comments":[{"comment":"The bottom rows of Table 3 (Micro/Macro/Weighted F-Score) are on a 0-100 scale, while Table 4 and Section 5.1.4 use 0-1 values; the units are never stated. This makes the tables hard to compare and may confuse readers.","section":"Table 3, Table 4"},{"comment":"Section 5.2.2 says the reasoning sample was \"LLAMA-generated responses at T0 using ZS,\" but Table 4 lists the top Weighted-F1 LLAMA configuration as T1 ZS. Clarify which configuration was used for the reasoning analysis.","section":"§5.2.2, Table 4"},{"comment":"The aggregate means are reported inconsistently: Section 5.1.5 gives µ=0.58 (binary) and µ=0.34 (multi-label), Table 6 reports means 0.56 and 0.35, and Table 2's bottom row shows µ_c=.57. The same aggregate statistics should be presented in one consistent way.","section":"§5.1.5, Table 2, Table 6"},{"comment":"Minor typos and notation issues: \"allowe\" (§5.1.2), \"Delta\" for ∆ (§5.1.5), and inconsistent use of plus signs in Table 6 (e.g., GPT-4.1 row \"+0.55\" despite ∆ defined as µ_b - µ_m).","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper's central result depends on the authors' own prior dataset [9], and the manuscript does not currently provide independent validation or inter-annotator reliability. This is not a reason to reject, but the editor may want to consider whether the artifact should be independently audited before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the granularity gap is probably real as a trend, but this paper does not have the statistical power to support the 41% number, and the abstract and body disagree on at least two points.\n\nWhat's genuinely useful: the line-level annotated dataset of 370 malicious PyPI packages with a 47-indicator taxonomy (from their prior work) is a solid resource, and the evaluation covers 13 models, five prompt strategies, three temperatures, and repeated trials. The reasoning analysis for LLAMA — 100 sampled responses with concrete examples of pattern matching, heuristics, and confidence calibration — is a nice qualitative contribution. The coder-versus-general split is also worth taking seriously.\n\nSoft spots, in decreasing severity. First, the binary task runs on 10 malicious packages. GPT-4.1's near-perfect F1 and the aggregate mean 0.58 are computed on that sample; the confidence interval is huge, and the paper itself concedes memorization is possible. Second, the multi-label task uses configuration-specific 37-file subsets, so the weighted-F1 means are averages over different test sets. Subtracting those two numbers to get a 41% decline is not statistically meaningful. Third, there is no baseline against existing static/dynamic scanners, so the 'use LLMs for triage' recommendation has no comparative support. Fourth, the malicious-package artifacts are not public, so independent replication is impossible. Fifth, the abstract's 'longer packages are harder' directly contradicts §5.3.5 (ρ = −0.04), and the abstract's 0.69/0.48 weighted-F1 numbers do not appear in the body. Finally, the granularity gap as a concept conflates task difficulty with capability: binary F1 and multi-label F1 are not directly comparable, and a meaningful gap needs a calibrated baseline.\n\nI do not think the qualitative direction is wrong. The item-level scores in Table 3 and the reasoning examples support the claim that models flag packages but miss context-dependent indicators. But the size of the gap, the 'near-perfect' GPT-4.1 figure, and the claim that only code complexity matters should all be treated as unreliable.\n\nWho this is for: security practitioners deciding whether to use LLMs as a first-pass filter, and researchers evaluating LLMs on code. It deserves a serious referee — the dataset and benchmark are worth engaging with — but the review should require confidence intervals, baseline comparisons, artifact release (or at least a public benchmark), and a fix for the abstract-body contradictions. In current form I would call it a promising draft that needs major revisions, not a paper whose numbers can be cited.","headline":"The granularity gap is probably real as a qualitative trend, but the 41% figure, the near-perfect GPT-4.1 score, and the 'only complexity matters' claim are not supported by the sampling, and the abstract contradicts the body.","tokens_in":28201,"tokens_out":5394,"would_cite":false,"duration_ms":52580,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that today's LLMs flag malicious PyPI packages well but lose roughly 41% of that accuracy when asked to pinpoint which lines are malicious.","keywords":["malicious packages","PyPI","LLM evaluation","granularity gap","indicator identification","multi-label classification","software supply chain security","prompt strategies"],"falsifier":"Re-run Task 2 on newly published malicious PyPI packages released after the models' training cutoffs, with line-level labels independently assigned by a second annotation team; if weighted F1 for indicators approaches the binary F1 on that fresh set, the granularity gap collapses.","tokens_in":27368,"feed_emoji":"🐍","tokens_out":5209,"duration_ms":50064,"temperature":0.7,"pith_summary":"The paper benchmarks 13 large language models on 4,070 PyPI packages (370 malicious, 3,700 benign) using two tasks: deciding whether a package is malicious, and identifying the specific malicious indicators inside it against a line-level taxonomy of 47 indicators. It reports a 'granularity gap': GPT-4.1 reaches near-perfect binary detection (F1≈0.99), but weighted F1 drops by roughly 41% for indicator identification. The main reason, per the paper, is that models detect syntactically rigid patterns (shell commands, eval/exec calls) yet miss intent-dependent behaviors such as import-time execution or dynamic module import, which look harmless in isolation. The paper recommends using LLMs for first-pass triage and human review for precise attribution; it also finds parameter size, context width, prompt strategy, and temperature have negligible or weak association with indicator-level accuracy. A sympathetic reader is meant to accept that current LLMs are powerful package-level filters but lack the semantic depth needed to explain or localize malicious behavior.","feed_headline":"LLM detection drops 41% when pinpointing malicious code lines","feed_subtitle":"Models flag malicious PyPI packages almost perfectly but can't reliably say which lines are the harmful ones.","key_machinery":"A two-stage benchmark built on a line-level annotated dataset: 370 malicious PyPI packages labeled with 47 malicious indicators in 7 categories, plus 3,700 benign packages. The 'granularity gap' is the operational identity: the difference between binary F1 (package-level detection) and weighted F1 (indicator-level identification). The paper uses five prompting strategies (zero-shot, few-shot, chain-of-thought, self-consistency, tree-of-thought), three temperatures, and Spearman correlations to test whether scale, context, or configuration explains performance; the concept of 'syntactic rigidity' carries much of the explanatory work.","core_discovery":"The central claim is the 'granularity gap': LLM performance collapses when moving from whole-package detection to fine-grained indicator identification. In the binary task, aggregate F1 is around 0.58 and GPT-4.1 reaches 0.99; in the multi-label task, aggregate weighted F1 drops to about 0.34, a decline of roughly 41%. The paper interprets this as evidence of shallow understanding: models often flag packages using surface-level heuristics without decomposing the code logic. Detection is best for indicators with fixed signatures (syntactic rigidity) and worst for behaviors whose harm depends on developer intent or broader context (e.g., import-time execution near zero). Coder-specialized mode","pith_inferences":["Inference: The granularity gap may be partly an artifact of how the ground-truth labels are defined; if indicator labels were restricted to purely syntactic signals, model performance would rise and the gap would shrink, while intent-labeled indicators keep the gap wide.","Inference: A testable extension is to combine LLM triage with static/dynamic analysis (e.g., taint tracking or sandboxing) aimed specifically at the near-zero indicators, predicting that hybrid pipelines recover most of the missing 41%.","Inference: The 'syntactic rigidity' explanation implies that obfuscating a rigid pattern (e.g., building os.system calls from strings) should push even coder models toward the near-zero group; this is a concrete prediction for future benchmarks.","Inference: The paper's scope is PyPI only; the authors note Python patterns differ from compiled languages, so the size of the granularity gap in npm, Maven, or RubyGems remains an open empirical question."],"forward_implications":["LLMs should be deployed as an initial triage layer that flags suspicious packages for human review, not as standalone detectors that assign specific malicious mechanisms.","If the gap is real, package-repository defenses that rely on LLM verdicts alone will inherit a blind spot for intent-dependent indicators such as import-time execution and dynamic module loading.","Coder-specialized models are the better choice when the threat has a rigid code structure (droppers, miners, shell commands), while general-purpose models are better for binary filtering and for behaviors requiring broad external knowledge.","Given the weak correlations with parameter count, context width, prompt strategy, and temperature, scaling up models or engineering prompts is unlikely by itself to close the granularity gap.","Because the gap is largest for the strongest binary detector (GPT-4.1), high package-level F1 should not be read as evidence of reliable malicious-intent reasoning."],"fun_headline_variants":["LLMs flag malicious packages but can't pinpoint harmful lines","41% accuracy drop when LLMs pinpoint malicious code","LLM detection falls apart on line-level malware clues","Models spot malicious PyPI packages, miss the smoking gun","Granularity gap: LLMs detect malware, not the malicious lines"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the 47-indicator, line-level ground-truth labels from the earlier dataset are correct and complete; as Section 6 acknowledges, if any labels are wrong or missing, every indicator-level F1 score and the measured granularity gap shift.","fun_headline_variants_meta":{"raw":{"variants":["LLMs flag malicious packages but can't pinpoint harmful lines","41% accuracy drop when LLMs pinpoint malicious code","LLM detection falls apart on line-level malware clues","Models spot malicious PyPI packages, miss the smoking gun","Granularity gap: LLMs detect malware, not the malicious lines"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000143,"raw_usage":{"total_tokens":1020,"prompt_tokens":766,"completion_tokens":254,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":510,"completion_tokens_details":{"reasoning_tokens":186}},"tokens_in":510,"tokens_out":254,"duration_ms":3193,"temperature":1.0,"reasoning_tokens":186,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T22:34:57.918945+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run Task 2 on newly published malicious PyPI packages released after the models' training cutoffs, with line-level labels independently assigned by a second annotation team; if weighted F1 for indicators approaches the binary F1 on that fresh set, the granularity gap collapses.","supporting_citations":[],"review_version":1}