{"id":"44deccd8-b65b-4118-b03e-223a43c18e55","arxiv_id":"2608.03232","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"MalTotal combines LLM-based sensitive API discovery, code slicing, and LLM judgment to detect malicious repositories across five languages with an average F1 of 93.1% at low cost per scan.","lead":"MalTotal uses a large language model to spot dangerous code in open source repositories across five programming languages, then trims the code down to the suspicious parts before asking the model to judge it. The system is claimed to be 94% cheaper per analysis than reading full files, and it found 564 malicious repositories in a scan of 120,000 GitHub projects.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Immediate-body-only TPL analysis has no false-negative evaluation; wrapper APIs can hide sensitivity in transitive callees and evade detection.","rationale":"The central claim is that MalTotal provides cost-effective, language-agnostic detection of malicious code. A necessary condition is that its sensitive API extractor reliably identifies sensitive operations, including those in third-party libraries. The paper deliberately limits TPL extraction to immediate bodies to save tokens. This creates a concrete blind spot: malicious wrappers that delegate to transitive callees are invisible to the extractor. The reader's weakest assumption points exactly here. This is not merely an external consistency concern; it is an internal design choice that is not validated against false negatives. The paper's manual review of TPL flags only addresses false positives, and no experiment assesses whether transitive-level sensitivity is commonly missed. The reported F1 on Multi-Lang-Bench could still be correct for that benchmark, so the verdict remains CONDITIONAL rather than REJECT. The reader already conditioned acceptance on addressing this and related evaluation issues; our analysis agrees and does not change the verdict. A secondary concern is the apparent tuning of hyperparameters (k=3, tau=0.3) on the evaluation set, which could bias the absolute F1, but the TPL immediate-body gap is the more security-specific and load-bearing issue for the system's stated purpose.","tokens_in":26459,"tokens_out":6582,"duration_ms":77299,"concrete_test":"Construct an adversarial benchmark of 100 malicious packages where the payload is reached only via a chain of TPL calls (e.g., pkg.trigger() calls lib._run(), which invokes os.system). Run MalTotal with its current TPL analysis and measure recall. If recall drops materially below the reported ~90%, the immediate-body restriction is a real blind spot. Alternatively, for a random sample of 200 cached TPL APIs, recursively analyze the full transitive closure of their call graphs and compare the set of sensitive APIs found; if additional sensitive APIs appear, the cache is incomplete.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In §3.1.1, MalTotal restricts third-party API analysis to the immediate function body, explicitly 'without recursively tracing its transitive callees.' This is load-bearing because the entire pipeline triggers only on a sensitive API call: if the LLM does not flag a TPL function as sensitive because the dangerous operation is hidden inside a transitive callee (e.g., wrapper calls helper that calls os.system), then the parsing-based filter never marks the call site, the slicer never runs, and the malware is missed. The paper's manual verification in §4.4 checks only false positives among the 347 flagged-sensitive TPL APIs, not false negatives. No experiment measures how often sensitivity is visible only beyond the immediate body. The ablation (w/o TPL) shows TPL identification helps, but the specific immediate-body design remains unvalidated. This is not contradicted by the benchmark F1, since the benchmark may not contain such wrappers, but it weakens the central claim of cost-effective, robust detection against real-world TPL-based evasion.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"MalTotal proposes a language-agnostic, LLM-assisted framework for detecting malicious code in open-source repositories. The pipeline consists of: (i) an LLM-assisted sensitive API extractor covering both built-in APIs (offline knowledge base) and third-party library (TPL) APIs (online analysis), (ii) a parsing-based pre-filter using Semgrep, (iii) a hybrid backward slicer on Joern Code Property Graphs that incorporates call-chain, control-dependency, and data-dependency slices with a call-depth limit k and Jaccard aggregation threshold tau, and (iv) an LLM-as-a-Judge semantic analyzer. The paper reports an average F1 of 93.1% across Python, JavaScript, Go, Java, and PHP on a new benchmark, a 94.0% token reduction versus file-level analysis, and a large-scale deployment on 120K GitHub repositories that discovered 564 previously unknown malicious repositories at a total cost of $338. The evaluation includes ablation studies, sensitivity analyses for slicing components, an LLM-comparison, a data-contamination check using post-cutoff packages, and a manual verification of TPL API identifications.","tokens_in":26703,"tokens_out":4179,"duration_ms":45516,"significance":"If the empirical claims hold, the paper makes a strong contribution: a scalable, multi-language detection pipeline with a compelling cost profile. The strengths are the public artifact (source code and data are promised), the extensive comparison against 8 baselines, the ablations that isolate the contribution of TPL identification and slicing, the sensitivity analysis on call depth, and the deliberate attempt to address data contamination with a post-cutoff test. The large-scale deployment, even with its limitations, is a meaningful practical demonstration. The main weakness is that a load-bearing design simplification — restricting TPL analysis to the immediate function body — is not validated against false negatives, and several evaluation choices (hyperparameter selection, small positive samples, no confidence intervals) leave room for the headline numbers to be optimistic. These issues are fixable with additional experiments rather than requiring a fundamentally different approach.","major_comments":[{"comment":"The TPL sensitive API collection restricts extraction to the immediate function body, explicitly \"without recursively tracing its transitive callees\" (§3.1.1). The entire pipeline depends on the LLM flagging a TPL function as sensitive before the parser emits rules and the slicer runs. A malicious wrapper that delegates a dangerous operation to a transitive callee (e.g., wrapper calls helper that calls os.system) is therefore invisible to the system. The manual verification (§4.4) samples 347 flagged-sensitive TPL APIs and measures only false positives, not false negatives; the w/o-TPL ablation (Table 6) shows TPL identification improves recall but does not validate the immediate-body restriction. No experiment measures how often sensitivity is visible only beyond the immediate body. This is load-bearing for the robustness claim against real-world TPL-based evasion. Please add a targeted","section":"§3.1.1 and §4.4"},{"comment":"The free parameters k (call depth, default 3) and tau (Jaccard threshold, default 0.3) are tuned empirically. Table 7 shows k=3 chosen on a 500-sample sensitivity set; no sensitivity analysis for tau is provided, and there is no held-out set to prevent selection overfitting. Since the reported average F1=93.1% is obtained under these choices, the benchmark numbers may reflect optimistic tuning. Please provide a tau sensitivity analysis and validate the chosen parameters on a separate holdout (or via nested cross-validation).","section":"§3.2 and Table 7"},{"comment":"For Java, Go, and PHP the positive samples are 75, 80, and 143 respectively (Table 4). The F1 differences between MalTotal and the next best (e.g., Java 92.3 vs 48.0 for AppInspector) are large, but the absolute F1 values and cross-model comparisons (Table 9) lack confidence intervals. With sample sizes this small, the reported cross-language F1 differences could be within noise; for example, DeepSeek-V3 vs GPT-4o on Go (91.5 vs 93.3) is likely not significant. Please report CIs (e.g., bootstrap) or apply statistical tests for the headline comparisons.","section":"Tables 4 and 5"},{"comment":"The large-scale study finds 564 \"confirmed previously unknown\" malicious repositories, but the confirmation is solely by the research team, with no inter-rater reliability statistic or independent external verification, and the sample population is the StarScout anomalous-repository list, which is not representative of GitHub as a whole. The reported FDR of 32.2% on the flagged set indicates substantial noise. Please provide inter-rater agreement (e.g., Cohen's kappa), a clear audit trail, and state explicitly how generalizable the discovery rate is outside the anomalous-popularity population.","section":"§4.5"}],"minor_comments":[{"comment":"The section title contains a typo: \"Evalution\" should be \"Evaluation\".","section":"Section 4"},{"comment":"The sentence \"Finally, our reliance on LLMs introduces a new attack surface.\" appears twice in consecutive paragraphs; remove the duplicate.","section":"§5"},{"comment":"The table uses the abbreviation \"OOC (%)\" while the text refers to \"N.A.\" samples; align terminology.","section":"Table 1 and §2.2"},{"comment":"MalTracker is described as \"JavaScrip only\" — typo for \"JavaScript\".","section":"§4.1"},{"comment":"The text states 27,746 TPL APIs were analyzed and 694 are sensitive; Table 3 sums to 694. Please clarify whether the 27,746 includes all analyzed functions (sensitive or not) and how caching affects these counts.","section":"§3.1.1 and Table 3"},{"comment":"The after-cutoff contamination test uses only 260 packages and reports no confidence intervals; the GPT-4o F1 drop (−4.7 and −3.9) is described as \"acceptable\" but is not tested for significance. Adding a small statistical comparison would strengthen the claim.","section":"Table 10"}],"recommendation":"major_revision","confidential_remarks":"The paper is technically solid and the empirical study is extensive, but the main gap is the unvalidated TPL immediate-body restriction, which is load-bearing for the robustness claim. I would request additional experiments on wrapper-based evasion and a proper holdout for hyperparameter selection before acceptance. The small language-specific sample sizes also need statistical grounding. These are fixable within the scope of a revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a real system, worth a serious referee. The contribution is not any single component—sensitive API lists, slicing, and LLM judging all exist—but the specific combination plus the scale of the evaluation. They ship code and data, and the measured 94% token reduction and $338 for 120K repos are concrete, reproducible cost claims. The post-cutoff contamination check on 260 packages is a good-faith attempt and mostly reassuring.\n\nThe soft spots are real but not fatal. First, k=3 and tau=0.3 are chosen from the same benchmark used to report final F1. Table 7 shows the performance peak at k=3; that's hyperparameter tuning on the test set. A separate validation split or nested CV would change the headline numbers by a couple of points, and reviewers should ask for it. Second, the Go/Java/PHP evaluation rests on 75–143 positives per language, with no confidence intervals; the cross-language generalization claim is plausible but the evidence is thin. Third, the RQ4 manual review has no inter-rater reliability numbers, so the 564 figure is a best-effort count, not a measured precision.\n\nThe issue I want to underline is the one the stress-test flags. In §3.1.1 the TPL analyzer restricts itself to the immediate function body, explicitly avoiding transitive callees. The paper's manual verification only samples flagged TPL APIs, so it has no false-negative measurement for this step. If a TPL function is a wrapper that calls a helper which eventually calls exec or os.system, the LLM never sees the dangerous operation, the API is never marked sensitive, and the whole pipeline never triggers. That is load-bearing. The ablation shows that TPL identification helps overall, but it does not validate the immediate-body simplification. This needs either a design change (trace one more level, or let the LLM see call targets) or a concrete study of how often sensitivity is visible only beyond the immediate body.\n\nWho is the audience: security researchers and practitioners working on supply-chain scanning. The benchmark and cost model are the most citable parts. I would send this to peer review, not desk-reject. The revision request should be specific: separate validation set, confidence intervals, and direct evidence about transitive-callee coverage in TPL extraction.","headline":"A genuinely useful scanner with real cost data and a credible large-scale deployment, but the current draft tunes hyperparameters on the test set and never tests a load-bearing false-negative gap in its TPL extraction; send it out with specific revision requests.","tokens_in":27156,"tokens_out":3721,"would_cite":true,"duration_ms":41725,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that reducing code to minimal, dependency-preserving contexts around sensitive API calls lets LLMs detect malicious code across languages at a small fraction of the usual token cost.","keywords":["supply chain security","malicious code detection","LLM-assisted analysis","code slicing","open source software","language-agnostic detection","code poisoning","static analysis"],"falsifier":"Build a small corpus of ground-truth malicious packages where each sample invokes a third-party function whose immediate body is benign but whose transitive callee performs an obvious sensitive action (process spawning or network exfiltration). Run MalTotal's pipeline on that corpus; if recall collapses while recall on directly-sensitive wrappers stays high, the immediate-body simplification is the specific limiting factor predicted by the described design.","tokens_in":26388,"feed_emoji":"🛡️","tokens_out":7491,"duration_ms":75899,"temperature":0.7,"pith_summary":"MalTotal claims that malicious code in open-source repositories can be detected at large scale, in many programming languages, by giving a large language model small focused slices of code instead of whole files or repositories. The system first finds sensitive APIs, including third-party library functions whose semantics are summarized and cached by an LLM, then slices backward from those API call sites along control, data, and call dependencies to build compact behavior contexts, and finally asks an LLM to judge intent. In a pilot study, whole-repository inputs failed or misled the LLM for up to 46 percent of samples, while slice inputs removed those failures and cut token cost by 97.6 percent. On a five-language benchmark the paper reports an average F1-score of 93.1 percent against eight baselines, and a 94 percent token reduction against file-by-file analysis. Applied to 120,000 GitHub repositories with over 7.3 million files, the pipeline cost about $338 and surfaced 564 previously unknown malicious repositories.","feed_headline":"Sliced-code LLM scans find 564 hidden malware repos for $338","feed_subtitle":"MalTotal trims LLM token use by 94 percent while detecting malicious code across five programming languages.","key_machinery":"The central mechanism is the hybrid backward slicer operating on a Code Property Graph (CPG), which jointly encodes abstract syntax, control flow, and program dependence. For each sensitive API call, the slicer traverses the CPG backward along call chains, control dependencies, and data dependencies to collect all code that could influence the call, with the call depth capped at three and entire callee bodies retained; then slices are de-duplicated by maximal-content filtering and merged by line-level Jaccard similarity. A second supporting mechanism is the LLM-assisted third-party API analyzer, which reads only the immediate function body of each library API to decide whether it is sensitiv","core_discovery":"MalTotal's central claim is that a three-stage pipeline — sensitive-API extraction, hybrid backward slicing over a Code Property Graph, and LLM-based semantic judgment — can identify malicious behavior in a language-agnostic way and at a cost that scales to millions of repositories. The extractor combines a curated knowledge base of 2,147 built-in sensitive APIs with an LLM that analyzes third-party library functions and caches semantic summaries; the slicer starts from each detected sensitive sink and follows call, control, and data dependencies up to depth three, keeping whole function bodies and merging overlapping slices into a minimal set of contexts; the judge classifies each context w","pith_inferences":["Editorial inference: the immediate-body-only rule for third-party APIs is the most likely failure point; a malicious wrapper that delegates to a sensitive transitive callee would slip past the extractor, and a targeted test corpus of such wrappers would clarify how much recall is being left on the table.","Editorial inference: the parameter choices (depth three, Jaccard threshold 0.3) are tuned on a small sample; checking whether the same settings hold for other languages or larger repositories would show how far the cost-accuracy balance generalizes.","Editorial inference: the LLM judge is itself an attack surface, and the paper's acknowledged prompt-injection risk suggests an A/B test of instruction-shielded prompts against the current prompt template.","Editorial inference: the headline economics ($338 for 120K repositories) count runtime inference for cached third-party APIs but not the one-time construction of the knowledge base and the 27,746 API analyses that produced the cache; a full lifecycle cost would matter for a security team adopting the pipeline."],"forward_implications":["If the results hold, malicious-code detection shifts from ecosystem-specific signatures to a generic semantic pipeline that can be ported to a new language by adding a parsing backend and a starter set of built-in sensitive APIs.","Token budgets cease to be the binding constraint on large-scale scans: a 94 percent reduction in LLM input makes a million-repository sweep economically plausible.","Detectors that ignore third-party library behavior miss a meaningful share of evasive malware; LLM-summarized library APIs recover part of that blind spot.","LLM verdicts come with machine-readable rationale (category, severity, analysis), so triage and false-positive review can be partly automated.","Because the method is sink-driven, it naturally suppresses false positives from benign code that merely imports sensitive libraries but never reaches a sensitive operation."],"supporting_citations":[{"why":"Supplies the Python and JavaScript evaluation corpus (MalwareBench) used to measure effectiveness.","marker":"[29]"},{"why":"Supplies the Java, Go, and PHP malicious-repository samples that become part of Multi-Lang-Bench.","marker":"[45]"},{"why":"SpiderScan supports the design of third-party-library-aware sensitive API identification and the restricted immediate-body extraction.","marker":"[22]"},{"why":"MalOSS provides the initial multi-language sensitive API specification that seeds the built-in knowledge base.","marker":"[6]"},{"why":"Adds JavaScript sensitive API specifications to the built-in knowledge base.","marker":"[68]"},{"why":"StarScout provides the high-risk repository population from which GitHub-120K is drawn for the large-scale study.","marker":"[19]"},{"why":"Joern generates the Code Property Graphs on which the hybrid backward slicer operates.","marker":"[26]"},{"why":"Defines the Code Property Graph intermediate representation that the slicing stage relies on.","marker":"[25]"},{"why":"DeepSeek-V3 is the LLM used for third-party API summarization and final maliciousness judgment, and its pricing underlies the cost estimates.","marker":"[5]"}],"fun_headline_variants":["MalTotal finds 564 hidden malware repos for $338","Cut LLM tokens 94%: uncovers 564 malware repos in 5 languages","Scan 120K repos for $338, detect 564 malicious ones","MalTotal: 93% F1, 94% lower cost, 564 malware catches","One tool, five languages: 564 malware repos at $338 total"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The load-bearing assumption is that reading only the immediate body of a third-party library API is enough to know whether that API is sensitive; if a malicious wrapper hides the sensitive operation behind another function call inside its body, the API is never flagged and the malicious context is never built.","fun_headline_variants_meta":{"raw":{"variants":["MalTotal finds 564 hidden malware repos for $338","Cut LLM tokens 94%: uncovers 564 malware repos in 5 languages","Scan 120K repos for $338, detect 564 malicious ones","MalTotal: 93% F1, 94% lower cost, 564 malware catches","One tool, five languages: 564 malware repos at $338 total"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000235,"raw_usage":{"total_tokens":1347,"prompt_tokens":761,"completion_tokens":586,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":505,"completion_tokens_details":{"reasoning_tokens":484}},"tokens_in":505,"tokens_out":586,"duration_ms":6777,"temperature":1.0,"reasoning_tokens":484,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T22:58:03.674114+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build a small corpus of ground-truth malicious packages where each sample invokes a third-party function whose immediate body is benign but whose transitive callee performs an obvious sensitive action (process spawning or network exfiltration). Run MalTotal's pipeline on that corpus; if recall collapses while recall on directly-sensitive wrappers stays high, the immediate-body simplification is the specific limiting factor predicted by the described design.","supporting_citations":[{"cited_title":"LLMs Caught in the Crossfire: Malware Requests and Jailbreak Challenges","cited_arxiv_id":"2506.10022","evidence_quote":"Supplies the Python and JavaScript evaluation corpus (MalwareBench) used to measure effectiveness."},{"cited_title":"Papalexakis, and Michalis Faloutsos","cited_arxiv_id":null,"evidence_quote":"Supplies the Java, Go, and PHP malicious-repository samples that become part of Multi-Lang-Bench."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"MalOSS provides the initial multi-language sensitive API specification that seeds the built-in knowledge base."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"StarScout provides the high-risk repository population from which GitHub-120K is drawn for the large-scale study."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Joern generates the Code Property Graphs on which the hybrid backward slicer operates."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the Code Property Graph intermediate representation that the slicing stage relies on."}],"review_version":1}