{"id":"72472468-f398-461c-aa71-fe1a371352c2","arxiv_id":"2509.14335","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A new evaluation framework shows current LLMs are not yet reliable at fine-grained Android malware behavior auditing, with the best model scoring 50.67% on the composite workload metric.","lead":"MalEval is a new benchmark that tests how well seven large language models can audit Android malware by explaining malicious behavior and backing it with code evidence. It finds that even the best model only reduces analyst workload by about half, and that models lean on surface clues instead of verifiable reasoning.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 50.67% WRS headline depends on unvalidated LLM-as-a-judge scores (RQ/EAS); if the judge is biased, the central quantitative claim and cross-model ranking do not measure auditing quality.","rationale":"The reader identifies the self-generated input confound in Section 3.4 as the weakest assumption, and that confound is real: comparing models when each model writes its own summaries and sensitivity scores does not cleanly isolate auditing ability. However, I see an even more load-bearing dependency: WRS and the headline '50.67%' are computed partly from RQ and EAS, which are produced by GPT-5-mini without any human-validated reliability check. The central claim about reliability levels is therefore only as strong as the judge, and the paper provides no evidence that the judge's scores correspond to expert judgments. The failure cases and behavior heatmaps are qualitative evidence and somewhat independent of the judge, but the quantitative WRS ranking and the '50.67%' figure are not. The reader's suggested fix (human-validated judge) and my proposed check align; the verdict should remain CONDITIONAL: accept only if the authors demonstrate judge reliability and address the input-generation confound or restrict conclusions to within-model comparisons.","tokens_in":22364,"tokens_out":1430,"duration_ms":12192,"concrete_test":"Take a stratified random sample of 50 generated reports across the seven models (e.g., 7-8 per model), have two independent malware analysts score the same RQ and EAS criteria used by GPT-5-mini on a 1-5 scale, then compute judge-vs-human Pearson/Spearman correlation and inter-rater reliability. If correlation is below 0.5 or the LLM judge's mean scores differ systematically by model, recompute WRS rankings with human scores and check whether Claude's top rank and the 50.67% figure survive.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central quantitative claim is the WRS across seven LLMs, with Claude at 50.67%. WRS is built from AA = product of RQ, EAS, SAS, where RQ and EAS are scored by GPT-5-mini as an LLM-as-a-judge without any human agreement study (Section 3.5.3, Eq. 3-5). If the judge systematically favors fluent, confident outputs over evidence-grounded ones, the WRS ranking and the conclusion that models 'rely on surface cues rather than verifiable evidence' could partly reflect judge bias rather than auditing ability. The reader's confound (each model generates its own D_c(f) and S(f) inputs in Section 3.4) is a genuine threat to cross-model fairness, but it compounds rather than replaces this concern: even with identical hand-written inputs, the reported RQ/EAS numbers would be unvalidated judge opinions. Moreover, SAS is a string-matching ratio that can be satisfied by citing any input function, regardless of whether that function actually supports the claimed behavior, so high SAS does not guarantee verifiability. A calibration study against human expert ratings is necessary: report Pearson/Spearman correlation and per-criterion agreement. The 50.67% WRS headline and comparative conclusions should be treated as conditional until such validation is provided.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"MalEval is presented as a diagnostic benchmark for fine-grained Android malware behavior auditing with LLMs. The paper curates 222 archived and 33 recent malware samples plus 25 recent benign apps with manually verified behavior reports and a 20,337-entry sensitive API list, reduces each app to reachable functions via static analysis, and represents functions as triples <signature, context-driven summary, sensitivity score>. It defines four analyst-aligned tasks—function prioritization (fidelity score), evidence attribution (CSR), behavioral synthesis (RQ/EAS/SAS), and sample discrimination (FPCR/TPMR/F1_c)—and aggregates them into a Workload Reduction Score (WRS). Seven LLMs are evaluated. The headline result is that the best model, Claude-3.7-sonnet, reaches only 50.67% WRS, with qualitative evidence that models rely on surface API cues, miss multi-step attack chains, and are sensitive to context formulation.","tokens_in":22778,"tokens_out":6130,"duration_ms":51664,"significance":"If the measurement pipeline is valid, MalEval would be a useful contribution: it provides a publicly released, reproducible benchmark with expert-verified ground truth at behavior and API level, decomposes auditing into separately diagnosable stages, and includes useful ablations (context removal in §4.3, metadata addition in §4.4) and a detailed failure-case analysis. The strongest parts are the dataset construction effort, the four-task decomposition, and the qualitative behavior-deviation heatmaps, which are less dependent on the judge model. However, the headline quantitative results (WRS values and cross-model rankings) currently rest on an unvalidated LLM-as-a-judge and on self-generated per-model inputs, so the quantitative claims should be treated as conditional until these are addressed.","major_comments":[{"comment":"RQ and EAS are assigned by GPT-5-mini as an LLM-as-a-judge (LLM Operations III and IV) with no human agreement study, inter-annotator statistics, or bias checks reported in Section 4. Because WRS in Eq. (3) multiplies RQ × EAS × SAS, the headline numbers in Table 2 (e.g., Claude 50.67%) and the rankings behind the 'surface cues' conclusion are partly opinions of one unvalidated judge. Please provide a calibration study (at least Pearson/Spearman correlation and per-criterion agreement against two or more human malware analysts on a sample of reports) and a check for judge bias toward fluent or confident wording.","section":"Section 3.5.3, Eq. (3)"},{"comment":"Each evaluated model generates its own context-driven summaries D_c(f) and sensitivity scores S(f), and those self-generated representations are then used as inputs to Tasks 1–4. The claim in §3.4 that this design 'ensures consistency across models' is not supported: a model that summarizes poorly incurs downstream penalties in FS, RQ, EAS, and SAS that are unrelated to its auditing ability, so the cross-model comparisons in Tables 2–4 are confounded. Please either evaluate all models on a fixed, model-independent intermediate representation or report both self-generated and fixed-input conditions.","section":"Section 3.4 (LLM Operation I)"},{"comment":"The Fidelity Score is computed by removing the top-10% most sensitive functions and measuring the confidence drop of an XGBoost classifier trained on TF-IDF features of function summaries, but the manuscript does not state whose summaries are used for training, nor does it report the classifier's accuracy or the stability of FS under different removal thresholds. Since FS differences drive the claim in §4.3 that inter-procedural context is indispensable, please report classifier accuracy and a threshold sweep.","section":"Section 3.5.1, Eq. (1)"},{"comment":"SAS is a syntactic existence ratio (|F_s|/|F_in|): it is satisfied by citing any function that appears in the input, regardless of whether that function supports the claimed behavior. The paper's conclusion that reports are 'verifiable' therefore rests on EAS, which is itself an unvalidated judge score. Please strengthen the grounding check (e.g., require cited functions to appear in the evidence chain for the behavior) or soften the verifiability claim.","section":"Section 3.5.3, SAS definition"}],"minor_comments":[{"comment":"Figure 1 contains the typo 'extreeAPK size' and Figure 2 contains 'Behevaior Report'; these should be corrected.","section":"Figures 1 and 2"},{"comment":"The ground-truth generation pipeline relies on an unspecified 'large language model' to parse vendor reports and to expand the sensitive API list; please name the model and version and describe any manual verification of the parsed output.","section":"Section 3.2.2"},{"comment":"The archived-versus-latest malware comparison uses Welch's t-test on very small subsets (33 latest samples across 3 categories); the statistical power is limited and the conclusions should be phrased more cautiously.","section":"Section 4.5.2"},{"comment":"WRS weights w_aa and w_ds are set equal without sensitivity analysis; please show WRS rankings under alternative weightings to demonstrate that the main conclusions are not weight-dependent.","section":"Eq. (5) and Section 4.1.2"},{"comment":"The heading 'Failure Case I' appears twice in the subsection; please renumber or merge the duplicated headings for clarity.","section":"Section 4.5.3"}],"recommendation":"major_revision","confidential_remarks":"The paper makes a solid empirical contribution, and the requested fixes (judge validation, fixed-input evaluation, FS classifier details) are feasible within the scope of a revision. The benchmark and qualitative failure analysis are valuable enough that the paper should not be rejected outright, but the quantitative claims should not be published in their current unvalidated form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, MalEval is a genuinely useful diagnostic benchmark for LLM-based Android malware auditing, and the qualitative finding—that current LLMs rely on surface cues, struggle to compose attack chains, and are sensitive to context—is plausible and supported by concrete failure cases. Second, the headline numbers, including Claude's 50.67% WRS, are not yet trustworthy enough to cite as quantitative fact. The paper deserves a serious referee, but it needs revision before the numbers can be used.\n\nWhat is actually new: the four-task decomposition (prioritization, evidence attribution, behavioral synthesis, sample discrimination) is a real extension beyond CAMA and MalLoc. The WRS composite is a sensible attempt to quantify analyst workload reduction. And the evidence-chain framing, where claims must be tied to functions, is the right problem to pose. The dataset, combining archived malware, post-2024 samples, and misclassified benign apps, is a genuine contribution. The ablation on inter-procedural context is informative, and the failure-case study (especially the CamScanner/Necro case) is concretely useful. Code and data are public.\n\nThe soft spots are real but manageable. The biggest is the input-generation confound in Section 3.4: each LLM writes its own summaries and sensitivity scores, so those become the inputs to all four tasks. A model that summarizes poorly receives worse inputs, so its downstream audit scores drop for reasons that are not about auditing ability. That weakens the cross-model rankings that drive the headline comparisons. The stress-test note is right that this compounds rather than replaces the judge problem: RQ and EAS are scored by GPT-5-mini with no human agreement study, and WRS multiplies those scores, so the central quantitative claim is only as solid as the judge's calibration. SAS is also weaker than it looks—it only checks that cited functions exist in the input, not that they support the claimed behavior. These are fixable: report human-judge correlation, and ideally fix inputs across models or at least show that results survive when the same summaries are used everywhere.\n\nThere are also visible copy-paste errors (the Failure Case IV generated text repeats Case III) and leftover ACM template artifacts. Minor, but sloppy in a version meant for review.\n\nBottom line: this is a serious paper with a valuable benchmark and a robust qualitative conclusion. The quantitative claims should be treated as conditional until the judge is validated and the cross-model confound is addressed, but the paper deserves full peer review. I would bring it to a reading group and would cite it, with a note that the numbers need confirmation.","headline":"A useful malware-auditing benchmark whose qualitative claims hold up, but whose headline numbers rest on unvalidated LLM judgment and a cross-model input confound.","tokens_in":699,"tokens_out":1774,"would_cite":true,"duration_ms":26662,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Current large language models cannot yet audit Android malware: the best of seven scores only 50.67% on the new MalEval benchmark, and models lean on surface cues instead of verifiable evidence.","keywords":["Android malware","malware behavior auditing","large language models","LLM evaluation","evidence attribution","static analysis","behavior synthesis","workload reduction"],"falsifier":"Re-run the four tasks with identical, externally authored function summaries and risk scores for every model, keeping all other settings fixed. If the model rankings on the Workload Reduction Score change materially, the reported cross-model differences are driven by summarization quality rather than by auditing ability; if the rankings hold, the benchmark measures what it claims.","tokens_in":22148,"feed_emoji":"🛡️","tokens_out":10812,"duration_ms":86294,"temperature":0.7,"pith_summary":"MalEval is a diagnostic benchmark for deciding whether large language models can do fine-grained Android malware auditing—not just flag a sample as malicious, but explain what it does and back that explanation with code-level evidence. The paper builds a dataset of real malware and misclassified benign apps with expert-written reports, compresses each app into function-level structural summaries, and splits auditing into four staged tasks that mirror analyst work. Seven widely used models are tested. The central result is that none is close to reliable: the strongest model reaches only 50.67 percent on the composite Workload Reduction Score, and the pattern of failures is consistent—models lean on surface cues, cannot fuse scattered facts into coherent attack chains, and change their judgments with context formulation. If this is right, security operations centers cannot yet delegate evidence-backed malware explanations to LLMs without substantial human oversight.","feed_headline":"LLMs fail malware auditing: top score 50.67%","feed_subtitle":"A four-stage benchmark shows LLMs lean on surface cues and cannot chain code evidence into attack narratives.","key_machinery":"The load-bearing object is the context-driven intermediate structural representation, written $\\mathrm{IR}(f)=\\langle N(f), D_c(f), S(f)\\rangle$: for every reachable function $f$, the signature $N(f)$, a summary $D_c(f)$ produced by the LLM from the function's one-hop caller–callee neighborhood, and a risk score $S(f)$. This representation is what makes traceable evaluation possible. Static reachability analysis from manifest and framework entrypoints first cuts the code space, then these structural units feed four staged tasks—function prioritization, evidence attribution, behavior synthesis, and sample discrimination—whose scores are combined into the Workload Reduction Score (WRS). The representation's role is to decouple audit judgments from raw, noisy codebases while keeping each claim anchored to a named function, which is what allows the benchmark to measure evidence grounding rather than free-form summarization.","core_discovery":"On the paper's own terms, the discovery is that current LLMs fail at exactly the reasoning that auditing requires. Given a context-driven structural representation of each function—signature, summary, and risk score—models can pick out some sensitive APIs and produce fluent narratives, but they rarely connect low-level evidence to the few decisive functions and behaviors that justify a malicious verdict. The paper reports that even the strongest of the seven evaluated models obtains a Workload Reduction Score of only 50.67 percent, and that behavior-level deviations are systematic: API-driven behaviors are over-observed, while reasoning-intensive behaviors such as privilege escalation and stealth are under-observed. Removing inter-procedural context lowers fidelity scores sharply, while adding application metadata helps some models but risks dropping true positives. The conclusion the paper draws is that LLMs are currently weak at verifiable, compositional evidence reasoning, so progress in malware auditing will require better-grounded and agentic workflows rather than isolated model outputs.","pith_inferences":["Editorial inference: a direct next step the paper leaves implicit is to force models to cite only actual signatures and to require cited functions to form connected call-graph paths; if the paper's diagnosis is right, this should improve evidence attribution and behavior synthesis more than scaling model size.","Editorial inference: the under-observation of privilege escalation and stealth across all models points to a shared inductive bias; minimal synthetic apps that isolate one behavior at a time could test whether models recognize these behaviors at all when API cues are neutral.","Editorial inference: the four stage-wise tasks double as training objectives, since each stage has a verifiable target; an auditing model could be fine-tuned with explicit evidence-grounding rewards rather than judged only on its final report."],"forward_implications":["If MalEval's results hold, current LLMs should not be the final authority in malware auditing; their reports need analyst verification, especially on evidence attribution and behavior synthesis.","The four-stage decomposition lets audit tools localize where an LLM fails, so only reliable stages—such as sensitive-API coverage—can be automated while human review remains on attack-chain reasoning.","Inter-procedural context is necessary for pinpointing decisive functions, since fidelity scores drop sharply when it is removed; future auditing prompts should preserve call relations.","Adding application metadata is not uniformly helpful: it can improve false-positive correction but can lower true-positive maintenance, and missing one malware sample can be catastrophic.","The framework's components—verifiable ground truth, context reduction, and structural representations—are platform-agnostic, so the evaluation can be adapted to Windows, Linux, and IoT malware."],"supporting_citations":[{"why":"Supplies the archived malware samples, the behavior categories, and the expert reports used as ground truth.","marker":"[47]"},{"why":"Provides the original sensitive-API list from which the benchmark's updated 20,337-API evidence list is built.","marker":"[5]"},{"why":"The closest prior benchmark for LLM Android malware analysis; the paper positions MalEval against its limits in verifiability and scale.","marker":"[22]"},{"why":"The earlier context-driven detection approach whose design informs MalEval's focused-signal context reduction.","marker":"[38]"},{"why":"Shows LLMs can attribute malicious behaviors to low-level methods with careful prompting; MalEval evaluates that capability at scale.","marker":"[42]"},{"why":"The study of analyst practice that motivates splitting auditing into four staged tasks.","marker":"[57]"},{"why":"Its method for detecting framework-overriding entrypoints is adapted for reachable-code extraction.","marker":"[59]"},{"why":"Supplies the fidelity-score approach used in Task 1 to measure whether removed functions drive classification.","marker":"[55]"},{"why":"Supports the LLM-as-a-judge protocol used to score report quality and evidence authenticity.","marker":"[16]"},{"why":"Documents failures in long-context code reasoning that motivate compressing codebases into structural representations.","marker":"[40]"}],"fun_headline_variants":["LLMs can't prove malware: top score 50.67%","Malware behavior audit: LLMs rely on surface cues, not proof","LLM malware audits: 50.67% ceiling, evidence gap","For malware auditing, LLMs narrate but can't justify","LLMs miss stealth, over-see APIs in malware audit"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Each evaluated model generates the function summaries and risk scores that later become its own test inputs, so a model that summarizes or scores poorly receives worse inputs and its audit score is depressed for reasons unrelated to auditing ability.","fun_headline_variants_meta":{"raw":{"variants":["LLMs can't prove malware: top score 50.67%","Malware behavior audit: LLMs rely on surface cues, not proof","LLM malware audits: 50.67% ceiling, evidence gap","For malware auditing, LLMs narrate but can't justify","LLMs miss stealth, over-see APIs in malware audit"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000595,"raw_usage":{"total_tokens":2837,"prompt_tokens":1050,"completion_tokens":1787,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":666,"completion_tokens_details":{"reasoning_tokens":1695}},"tokens_in":666,"tokens_out":1787,"duration_ms":11742,"temperature":1.0,"reasoning_tokens":1695,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T15:51:17.776586+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the four tasks with identical, externally authored function summaries and risk scores for every model, keeping all other settings fixed. If the model rankings on the Workload Reduction Score change materially, the reported cross-model differences are driven by summarization quality rather than by auditing ability; if the rankings hold, the benchmark measures what it claims.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the archived malware samples, the behavior categories, and the expert reports used as ground truth."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the original sensitive-API list from which the benchmark's updated 20,337-API evidence list is built."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The closest prior benchmark for LLM Android malware analysis; the paper positions MalEval against its limits in verifiability and scale."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The earlier context-driven detection approach whose design informs MalEval's focused-signal context reduction."},{"cited_title":"MalLoc: Toward Fine-grained Android Malicious Payload Localization via LLMs","cited_arxiv_id":"2508.17856","evidence_quote":"Shows LLMs can attribute malicious behaviors to low-level methods with careful prompting; MalEval evaluates that capability at scale."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The study of analyst practice that motivates splitting auditing into four staged tasks."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Its method for detecting framework-overriding entrypoints is adapted for reachable-code extraction."},{"cited_title":"Sense and Sensitivity: Examining the Influence of Semantic Recall on Long Context Code Understanding","cited_arxiv_id":"2505.13353","evidence_quote":"Documents failures in long-context code reasoning that motivate compressing codebases into structural representations."}],"review_version":2}