{"id":"13a833c5-93eb-42a7-af5f-18d2b082a909","arxiv_id":"2508.21579","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"An LLM-agent pipeline discovers Android app vulnerabilities and validates many of them with automatically generated proof-of-concept exploits.","lead":"A2 is an automated system that uses multiple AI agents to find security flaws in Android apps and then tries to prove each flaw by building a working attack. If the self-validation holds, it would shift Android security work from triaging thousands of warnings to reviewing a short list of exploit-backed findings.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 104 zero-day and 57 self-validated PoC claims rest on LLM-generated oracles with no external ground truth, so the counts are unsupported without independent confirmation.","rationale":"The paper's most defensible result is the Ghera detection and validation evaluation, because Ghera has external labels and the 78.3% coverage and 68.0% validation rates can in principle be checked against known ground truth. My concern targets the validation link itself: Section 3 defines a Validated Vulnerability Finding as one with 'an LLM assessment' that the PoC demonstrates the vulnerability, explicitly allowing false positives if validation is misleading. Section 4.3's oracle is generated by the validator LLM, so the apparent independence of validation is weaker than claimed. Table 7 provides direct evidence that the pipeline accepts false positives in the unified configuration (OpenSocket-InformationLeak-2, marked ⊙), and Section 5.5's own hallucination rates confirm the mechanism is imperfect. On Ghera the damage is bounded because labels exist; on the 160 production APKs there is no such bound, and the paper provides no vendor confirmations, no released artifacts, and withheld APK identifiers. The reader's weakest assumption identifies exactly this reliability premise, and the CONDITIONAL verdict already encodes the need for independent confirmation. I therefore agree with the reader and do not move the verdict. The proposed concrete test would settle the concern: if independent analysts reproduce the 57 self-validated PoCs and confirm exploitability, the central real-world claim is materially strengthened; if not, the 104 zero-day count should be revised. I note secondary internal inconsistencies (Table 4 vs Section 5.5 denominator language), but they are not the load-bearing issue compared with the absence of external ground truth for the zero-day claims.","tokens_in":26217,"tokens_out":7036,"duration_ms":65796,"concrete_test":"Obtain the original 169 AndroZoo APKs and the full A2 execution logs for the 57 self-validated findings, then have two independent security researchers who are not authors re-run each PoC on a clean emulator and independently classify exploitability against a pre-registered rubric (e.g., requires no special permissions, crosses a trust boundary, produces observable state change). Compare their labels with A2's 104 TP / 57 validated counts; if, on a random sample of 20 findings, any PoC fails to reproduce or any \"validated zero-day\" is independently judged non-exploitable, the zero-day counts should be revised downward and the paper's central claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that A2's \"validated\" findings are genuinely exploitable, but this is not established. In Section 4.3, the Task Validator generates its own oracles using the same LLM family that produced the original claims; the oracle is an LLM judgment, not an independent check. Section 5.5 reports residual hallucination rates of 12.6% (mixed) and 4.7% (unified) at the claim level, and Table 7 shows that in the unified configuration the known false positive OpenSocket-InformationLeak-2 is marked with ⊙ (false positive misclassified as true positive). Thus the validation pipeline demonstrably certifies false positives even on Ghera, where ground truth labels exist. For the 160 production APKs in Section 6, there is no external ground truth: the 104 true positives and 57 self-validated findings rest only on the authors' manual review, vendor confirmations are not reported, and Section 12 withholds APK identifiers and artifacts. If the validator's PASS decisions carry even the measured false-positive rate, the zero-day and self-validated-PoC counts are unsupported and the headline numbers reduce to unvalidated LLM output.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces A2, a two-phase LLM-agent system for Android vulnerability discovery and validation. The discovery phase combines four LLM analyzers with optional SAST tool signals and aggregates them into speculative vulnerability findings; the validation phase uses a Planner-Executor-Validator loop to generate PoCs and judge them via LLM-generated oracles. On the Ghera benchmark (n=60), A2 reports 78.3% coverage (47/60) across four aggregation models, and validated-PoC rates of 61.3% (mixed config) and 68.0% (unified config). On 160 production APKs (from 169 AndroZoo samples), the paper reports 104 true-positive zero-day vulnerabilities, 57 with self-validated PoCs, and claims significant reduction in warning volume relative to SAST tools.","tokens_in":26457,"tokens_out":8233,"duration_ms":73881,"significance":"If the validation claims held, this would be a substantial step toward end-to-end automated Android vulnerability assessment. The Ghera coverage result is genuinely anchored: 47/60 is reproduced by four different aggregation engines and exceeds the best individual model and all SAST baselines, and the cost/efficiency analysis is unusually detailed. The paper also honestly reports residual hallucination rates (12.6% mixed, 4.7% unified) and one documented false-positive-as-true-positive case. However, the central validation claim rests on LLM-generated oracles produced and judged by the same model family that issued the original findings; the paper's own data show this oracle can certify false positives, and the real-world zero-day counts have no external ground truth. The measured false-positive certification rate, the inconsistent denominator in the validated-rate computation, and the withheld real-world artifacts mean the headline 'validated' numbers are not yet established.","major_comments":[{"comment":"The validated finding rates 46/75 (61.3%) and 51/75 (68.0%) use a denominator of 75, which is 82 total findings minus 7 false positives. But the 7 FPs come from the 63 evaluated findings (82−2−17=63), while the 17 out-of-scope and 2 unable-to-install findings were explicitly excluded from evaluation. The denominator therefore includes 19 unevaluated findings, which overstates the number of actionable cases and understates the success rate. Please recompute the rates on the evaluated set of 63 findings (e.g., 46/63 and 51/63) and on the non-FP subset of those findings, and report both clearly.","section":"§5.5, Table 7"},{"comment":"The validation oracle is designed and executed by the Task Validator, an LLM drawn from the same Gemini 2.5 family that produced the speculative findings in the discovery phase, and the PASS/FAIL decision is an LLM judgment rather than an independent check. The manuscript's own measurements show this oracle certifies false positives: Table 7 lists OpenSocket-InformationLeak-2 as ⊙ (false positive misclassified as true positive) in the unified configuration, and §5.5 reports residual hallucination rates of 12.6% (mixed) and 4.7% (unified) among execution claims. Consequently, a 'validated vulnerability finding' as defined in §3 does not yet demonstrate exploitability with the reliability claimed in the abstract. Please add an independent validation component (e.g., human expert re-review of a random sample of PoCs, or deterministic oracles for cryptographic/token-based findings) and report per-finding whether the deciding oracle is deterministic or LLM-based.","section":"§4.3, §5.5, Table 7"},{"comment":"The 104 real-world zero-day vulnerabilities and 57 self-validated PoCs are labeled only by the authors' manual review; no vendor confirmations are reported, and §12 withholds application identifiers, sample names, and PoC artifacts. Because §5.5 establishes that the validation pipeline certifies false positives at a measurable rate (including a concrete ⊙ case on Ghera), the real-world counts are not independently verifiable as presented. Please provide access to PoC artifacts or detailed logs for reviewers, report vendor responses (even aggregated), or present an independent re-validation of a randomly selected subset of the 57 claimed self-validated findings.","section":"§6, §12"}],"minor_comments":[{"comment":"The abstract and Table 4 say 169 production APKs, while §6 says 9 APKs were excluded as uninstallable and the analysis covers 160 APKs; please clarify which number refers to the scanned set.","section":"§6, Table 4"},{"comment":"The table heading contains a typo: 'UnnecesaryPerms-PrivEscalation-1' should read 'UnnecessaryPerms-PrivEscalation-1'.","section":"Table 7"},{"comment":"The contribution bullet states that A2 'correctly rules out all false positives (7/7)', but Table 7 shows that the unified configuration misclassifies OpenSocket-InformationLeak-2 as a true positive (⊙); please specify that the 7/7 claim applies only to the mixed configuration.","section":"§5.5, Contribution list"},{"comment":"The detection efficiency metric E_i includes the unknown total vulnerability count V_total; since V_total is a constant across tools, the metric is only meaningful as the ratio R_i/A_i. Please state that V_total cancels in relative comparisons or remove it from the definition.","section":"§5.4, Eq. (1)"},{"comment":"Several references appear to be generic placeholders rather than verifiable sources (e.g., [20] 'John Smith, Jane Lee, and Chen Wang' and [23] 'Richard Chen, Sofia Martinez, and David Kim'). Please verify these citations and replace them with the actual published works.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper is otherwise well-structured and the Ghera coverage experiment is solid, but the validation and zero-day claims need substantially more support before publication. In addition to the technical issues in the report, I am concerned about the citation list: several entries appear to be fabricated or placeholder references, which is a serious scholarly integrity issue that the editor should check carefully. The real-world evaluation would be much stronger if the authors provided reviewers with access to the PoC artifacts or at least an independent third-party verification of a sample of the zero-day findings."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Punchline: A2 is a real engineering contribution, and the Ghera detection result is the credible core; the headline 104-zero-day claim is not supported as written because the validation loop is self-referential and the real-world labels have no external ground truth.\n\nWhat is actually new: the two-phase agentic pipeline—LLM discovery plus planner/executor/validator with 29 Android-specific tools—is, as far as these citations show, new for Android. The Ghera coverage number (47/60, 78.3%) is consistent across four aggregation models and beats the SAST tools tested. The consolidation from thousands of warnings to ~82 findings is a genuine practical gain. Credit where it's due: the paper explicitly reports residual hallucination rates of 4.7–12.6%, flags its own misclassified false positive (OpenSocket-InformationLeak-2 in unified config), and states the validation scope limitations. That transparency is welcome.\n\nSoft spots. First, the validator generates its own oracles using the same LLM family that produced the speculative findings. That is not an independent check; a hallucinated claim can get a hallucinated oracle that passes it. The Ghera experiment gives some external anchor: the pipeline is evaluated against known ground truth, and the 78.3% detection recall is meaningful. But the validation success rates (61–68%) are computed against the LLM's own judgments, not the Ghera labels. Second, the real-world zero-day counts rest on the authors' manual review of LLM output. No vendor confirmations are reported, APK identifiers are withheld, and artifacts are gated behind institutional email vetting. Reproducibility is therefore low. Third, there are internal inconsistencies: Table 4 reports validation TP=51 FP=1, while Table 7 (which the reader should trust more) shows 7 false positives in the source set and one ⊙ misclassified as TP. And the 61.3%/68.0% success rate denominators use 75 = 82−7, which includes 17 out-of-scope and 2 uninstallable cases, not the 63 actionable findings. That inflates or obscures the true rate. Fourth, each LLM run is executed once; no variance or rerun results.\n\nThe detection half of the paper holds up. The validation half needs external confirmation, artifact release, and numeric cleanup before the zero-day claims can be taken at face value.\n\nWho it's for: people working on LLM agents for security, or automated Android analysis. Worth a serious referee, but the review should push for artifact release, independent/vendor validation of a sample of zero-days, and corrected statistics. I'd accept it with major revision rather than reject.","headline":"A useful engineering paper whose Ghera detection core is credible; the headline 104-zero-day claim is not supported as written because the validation loop is self-referential and the real-world labels lack external ground truth.","tokens_in":26989,"tokens_out":4942,"would_cite":true,"duration_ms":44245,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A two-phase agentic pipeline both finds and validates Android vulnerabilities, covering 78.3% of a benchmark and finding 104 real-world zero-days.","keywords":["Android vulnerability detection","agentic security analysis","proof-of-concept generation","LLM agents","zero-day vulnerabilities","static application security testing","vulnerability oracle","emulator-based validation"],"falsifier":"Re-run the validation pipeline on a random sample of the 57 claimed self-validated zero-day PoCs with a different LLM as task validator and a fresh emulator, and record how many still pass; if the pass rate falls far below 100%, the validation signal is partly model-specific rather than a property of the exploits. A complementary check runs the same loop on a set of known-benign APKs and counts how many speculative findings wrongly pass oracle checks.","tokens_in":26008,"feed_emoji":"📱","tokens_out":7388,"duration_ms":56090,"temperature":0.7,"pith_summary":"The paper argues that a single AI-agent pipeline can do what static analyzers and fuzzers have not: actually prove that Android vulnerabilities are exploitable by generating working proof-of-concept attacks. The proposed system, A2, first combines LLM code reading with standard static-analysis tools to produce a short list of speculative vulnerabilities, then runs a planner–executor–validator loop that designs, executes, and independently verifies an exploit for each candidate. On the 60-app benchmark the pipeline covers 78.3% of labeled vulnerabilities, far above the 30.0% of the leading static tool, while emitting a few dozen findings instead of thousands of warnings. On 160 production apps it reports 104 previously unknown true-positive vulnerabilities, 57 of them accompanied by automatically generated and self-validated PoCs. The significance, if the results hold, is that automated security analysis can move from flagging suspicious code to demonstrating real attacker impact.","feed_headline":"Agentic pipeline finds 104 Android zero-days","feed_subtitle":"Two-phase system validates exploits with working PoCs, covering 78% of benchmark vulnerabilities.","key_machinery":"The load-bearing mechanism is the three-agent validation loop with LLM-generated oracles. For each speculative finding, a PoC Planner produces a task list with expected outcomes (using read-only tools only), a Task Executor carries out the plan through function calls, and a Task Validator—kept read-only and forbidden to trust the executor's self-report—constructs custom oracle checks per finding and re-runs the relevant steps to decide PASS or FAIL; failures return to the planner for replanning. The oracle step is the core novelty: instead of fixed heuristics, each expected effect becomes specific assertions (e.g., a file exists, a cryptographic key matches a value, the resumed activity equals the target), which the validator re-computes independently.","core_discovery":"A2's central claim is that vulnerability discovery and validation can be unified in one agentic workflow, and that the validation half—turning a suspicion into an executable exploit with independent verification—is what separates its results from prior tooling. In the discovery phase, LLMs analyze decompiled code and the manifest, optionally enriched by warnings from three static analyzers; an LLM aggregator deduplicates and filters these into a small set of speculative findings. In the validation phase, a PoC planner writes a step-by-step exploitation plan, a task executor executes it through 29 function calls spanning UI, intents, file system, logs, code execution, and custom APK builds, and a task validator independently re-checks each claimed effect using dynamically generated oracles and feeds failures back for replanning. The paper reports that on the 60-app benchmark the full loop validates 68.0% of actionable findings with its strongest configuration and correctly rejects all seven false positives it was given; on 160 production APKs it reports 104 true positives with 54.8% validated end-to-end by automatically generated PoCs.","pith_inferences":["If the oracle-based validation pattern generalizes, the same planner–executor–validator architecture, with domain-specific oracles, could transfer to iOS, web, or firmware security instead of remaining Android-specific.","The 57 self-validated real-world PoCs suggest a new success metric for vulnerability scanners — PoC yield — where a finding counts only when an executable and independently checked exploit exists; this would raise the bar for both static analyzers and fuzzers.","A decisive test not run in the paper is cross-model validation: re-validating a sample of the 57 PoCs with a different LLM family as the validator to see whether the oracle judgments are model-dependent.","The 5 MB APK size cutoff means large enterprise apps are currently out of reach; a testable extension is retrieval-based, hierarchical code analysis that would tell whether the coverage and validation gains persist on bigger codebases."],"forward_implications":["Static analysis tools retain a supporting role in A2: they provide code-line localization and call-graph evidence that guides and filters the agent, but they are no longer the primary detector since LLM analyzers already cover every vulnerability the static tools find on the benchmark.","The per-finding validation cost ($0.59 to $26.85 depending on model choice, median around $1.77 for the cheaper configuration) is low enough that automated PoC generation can sit inside a triage workflow instead of being a manual research task.","The reported residual hallucination rate of 4.7 to 12.6 percent for executor claims is a quantifiable floor for manual review: roughly one in twenty to one in eight steps an executor reports is wrong, so human review of validated findings remains necessary.","Seventeen vulnerability classes (notably network man-in-the-middle attacks) fall outside the validation scope because they are not observable as state changes on a stock emulator, which delimits the system's current attack surface.","Long-context function-calling degradation after about 14 execution steps motivates cycle detection that halts a stuck executor, reducing the observed cost blow-ups."],"supporting_citations":[{"why":"the benchmark of 60 vulnerable Android apps whose labeled ground truth defines coverage and validation success.","marker":"[32]"},{"why":"the static analyzer whose 30.0% coverage on the benchmark is the main baseline A2 surpasses.","marker":"[6]"},{"why":"a static tool included as a baseline and as a source of formatted warnings feeding the discovery phase.","marker":"[5]"},{"why":"a bytecode-level static analyzer used as a third baseline and integration signal.","marker":"[35]"},{"why":"the corpus of production APKs from which the real-world zero-day dataset is sampled.","marker":"[11]"},{"why":"the orchestration layer used to coordinate the planner, executor, and validator agents.","marker":"[24]"},{"why":"the decompiler that converts APK bytecode into source code for LLM analysis.","marker":"[25]"},{"why":"a prior end-to-end exploit-generation system whose oracle concept motivates the validator's design.","marker":"[31]"}],"fun_headline_variants":["Agentic system finds 104 Android zero-days","AI agent validates Android vulnerabilities with PoCs","Two-phase tool covers 78% of Android benchmark","Working exploits confirm 54% of discovered Android bugs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The results stand or fall on whether a validation step designed and judged by an LLM, fed back through another LLM, is real evidence of exploitability; if those judgments are systematically unreliable, the validated-finding counts and the zero-day numbers are unverified model output.","fun_headline_variants_meta":{"raw":{"variants":["Agentic system finds 104 Android zero-days","AI agent validates Android vulnerabilities with PoCs","Two-phase tool covers 78% of Android benchmark","Working exploits confirm 54% of discovered Android bugs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000287,"raw_usage":{"total_tokens":1737,"prompt_tokens":1047,"completion_tokens":690,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":663,"completion_tokens_details":{"reasoning_tokens":629}},"tokens_in":663,"tokens_out":690,"duration_ms":5676,"temperature":1.0,"reasoning_tokens":629,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:39:58.263868+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the validation pipeline on a random sample of the 57 claimed self-validated zero-day PoCs with a different LLM as task validator and a fresh emulator, and record how many still pass; if the pass rate falls far below 100%, the validation signal is partly model-specific rather than a property of the exploits. A complementary check runs the same loop on a set of known-benign APKs and counts how many speculative findings wrongly pass oracle checks.","supporting_citations":[{"cited_title":"https://secure-it-i.bitbucket","cited_arxiv_id":null,"evidence_quote":"the benchmark of 60 vulnerable Android apps whose labeled ground truth defines coverage and validation success."},{"cited_title":"APKHunt: A comprehensive static code analysis tool for android apps, 2024","cited_arxiv_id":null,"evidence_quote":"the static analyzer whose 30.0% coverage on the benchmark is the main baseline A2 surpasses."},{"cited_title":"MobSF: Mobile security framework,","cited_arxiv_id":null,"evidence_quote":"a static tool included as a baseline and as a source of formatted warnings feeding the discovery phase."},{"cited_title":"Trueseeing: Non-decompiling android application vulnerability scanner, 2024","cited_arxiv_id":null,"evidence_quote":"a bytecode-level static analyzer used as a third baseline and integration signal."},{"cited_title":"Bissyandé, Jacques Klein, and Yves Le Traon","cited_arxiv_id":null,"evidence_quote":"the corpus of production APKs from which the real-world zero-day dataset is sampled."},{"cited_title":"Langgraph","cited_arxiv_id":null,"evidence_quote":"the orchestration layer used to coordinate the planner, executor, and validator agents."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"the decompiler that converts APK bytecode into source code for LLM analysis."}],"review_version":2}