{"id":"985cccbf-7a6f-4275-b007-dd7b1d163d36","arxiv_id":"2504.18423","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"A proposed RAG plus mixture-of-agents LLM pipeline for Android vulnerability detection shows a small, manually labeled improvement on one toy app, without reproducible artifacts.","lead":"LLMpatronous combines retrieval-augmented generation (RAG) with a mixture of several large language models (MoA) to detect vulnerabilities in Android Java code. The three experiments on a single small vulnerable app report fewer false positives, but missing code, data, and ablations leave the central claim unsupported.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Experiment 3 cannot support the central claim because it re-scored Experiment 2's candidates with a different model set, no ablation isolates RAG or MoA, and one row was relabeled on plausibility.","rationale":"The reader's weakest_assumption pins the load-bearing issue exactly: the improvement in Experiment 3 is confounded by simultaneous changes in models, RAG, prompt, and scoring, with no ablation. My independent reading of Section IV confirms this. Experiment 1 already fails as a baseline: the text first reports 'approximately 4 out of the 8' specified vulnerabilities, then a subsequent note says the user's original interpretation of 7/8 'seems optimistic,' then entertains that 7/8 might hold—so the baseline accuracy figure is not settled. Experiment 2 correctly identifies 6 of the original 8 plus three additional candidates and one explicit false positive ('Insecure Design'), which is the only entity Experiment 3 claims to reject. Experiment 3 then verifies only the Experiment 2 candidate list, and Table V contains a documented post-hoc relabeling ('Hardcoded Credentials' changed from False to True 'based on plausibility, assuming MoA confirmed it'). That single admission means the paper's own evidence is not a completed experiment but a partially assumed one. The conclusion that RAG+MoA 'significantly mitigated the false positive problem' is therefore an overclaim relative to the evidence. I agree with the reader's REJECT verdict: the idea is plausible and the paper correctly identifies the right research question (hallucination-induced false positives), but the evaluation does not demonstrate that RAG+MoA causes the observed improvement, nor does it quantify any improvement at all. I would not soften the verdict to CONDITIONAL because the missing artifacts (no prompts, no RAG configuration, no per-agent outputs, no reproducible code despite the stated GitHub link) make the concern untestable by a reviewer without independently reimplementing the entire hand-written workflow, and the one table that carries the central claim contains an explicit post-hoc adjustment.","tokens_in":10278,"tokens_out":2951,"duration_ms":20665,"concrete_test":"Run a 2x2 ablation on Vuldroid with the same expanded 25-item list and the same MoA prompt template: (1) single Gemini 1.5 Pro with no RAG, (2) Gemini 1.5 Pro plus RAG context, (3) the open-source MoA chain with no RAG, (4) the open-source MoA chain plus RAG. Then re-run the Experiment 2 candidates through all four arms and compare the per-candidate pass/reject labels against Table V. If arm (2) alone already rejects 'Insecure Design' and confirms 'Hardcoded Credentials,' then RAG—not MoA—is doing the reported work; if arm (3) also rejects it, then the model change alone explains the result. If no arm reproduces Table V without the authors' plausibility adjustment, the central claim loses its experimental support.","verdict_should_be":"REJECT","load_bearing_attack":"The paper's central claim—that RAG+MoA 'significantly mitigated the false positive problem'—rests entirely on Experiment 3, but Experiment 3 does not test RAG+MoA against the same pipeline. Experiment 2 used a single Gemini 1.5 Pro prompt; Experiment 3 switched to an open-source MoA chain (Llama 3.1 70B, Qwen2 72B) and added RAG, changing the model, the prompt, and the scoring procedure at the same time. There is no ablation: no RAG-only arm, no MoA-only arm, no single-open-source-model-with-RAG arm, and no re-run of the Experiment 2 candidates with the Experiment 2 model but with RAG added. The one reported success—rejecting 'Insecure Design' as a false positive—could be due to the stronger open-source models, the RAG context, the MoA aggregation, the different prompt, or simply the authors' relabeling. In fact, the manuscript admits that Table V's 'Hardcoded Credentials' row was changed from False to True 'based on plausibility, assuming MoA confirmed it,' so the table is not a raw record of Experiment 3 output. With no quantitative metrics (only a small csv of per-candidate labels on one intentionally vulnerable app), no released prompts or configurations, and no SAST/DAST or basic-LLM baseline, the observed improvement cannot be attributed to the proposed architecture. The conclusion therefore overclaims what the experiments establish.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes LLMpatronous, a workflow that combines Retrieval-Augmented Generation (RAG) and a Mixture of Agents (MoA) to detect vulnerabilities in Android source code. The authors motivate the approach by the false-positive and hallucination problems of single-LLM prompting, and they evaluate it in three experiments on the intentionally vulnerable Vuldroid application: (1) a single Gemini 1.5 Pro call with a predefined list of eight vulnerability types; (2) the same setup with an expanded list of 25 types; and (3) an open-source MoA pipeline with RAG re-scoring the candidates produced in Experiment 2. The paper reports that Experiment 3 confirmed most of the Experiment 2 true positives and rejected the 'Insecure Design' false positive, and concludes that the RAG+MoA methodology significantly mitigated the false-positive problem. Sections V-VII discuss limitations and future work.","tokens_in":10516,"tokens_out":5905,"duration_ms":51894,"significance":"If properly supported, showing that RAG+MoA reduces false positives in LLM-based vulnerability detection while using open-source models would be a useful practical contribution to security tooling. The paper also gives a clear architecture description and an honest limitations section. However, the manuscript does not supply the required support: the evaluation is a set of three manual, small-scale experiments on one application; there are no quantitative metrics, no controlled comparisons, no ablations, and no released evaluation scripts. The central empirical claim therefore remains unsubstantiated in its present form.","major_comments":[{"comment":"Experiment 3 does not test the proposed RAG+MoA pipeline against the same pipeline as Experiment 2. Experiment 2 used a single Gemini 1.5 Pro call with one prompt; Experiment 3 switched to a different set of open-source models (Llama 3.1 70B, Qwen2 72B) in a sequential MoA chain, added RAG, and re-scored only the candidates already found in Experiment 2. The observed rejection of 'Insecure Design' and confirmation of the other candidates could be caused by the model swap, the RAG context, the MoA aggregation, the different prompt, or the re-scoring procedure. Without an ablation that isolates RAG or MoA (for example, RAG-only, MoA-only, or a single open-source model with RAG), the central claim that RAG+MoA 'significantly mitigated the false positive problem' is unsupported.","section":"Section IV-C, Table V"},{"comment":"One reported result is relabeled after the fact. The note under Table V states: 'The user's original table marked \"Hardcoded Credentials\" as False, but given Vuldroid's nature, it's highly likely present, and MoA should ideally confirm it if evidence exists. We adjust this based on plausibility, assuming MoA confirmed it.' Consequently Table V is not a record of the RAG+MoA output; it is a hybrid of raw output and the authors' plausibility judgment. The claim that Experiment 3 confirmed the true positives therefore does not follow from the experiment.","section":"Section IV-C, Table V note"},{"comment":"The paper's own report of the Experiment 1 result is internally inconsistent. The text says the LLM identified patterns related to 'approximately 4 out of the 8 specified vulnerabilities,' while the note says the original interpretation stated '7/8 identified'; Table II itself lists five 'Yes' matches. These numbers are not mutually consistent, and the discrepancy is not explained. This matters because Experiment 1 is the baseline against which the false-positive improvement is measured.","section":"Section IV-A, Table II"},{"comment":"The evaluation contains no quantitative metrics anywhere. No precision, recall, F1, false-positive rate, or confidence intervals are reported; the results are presented as a small set of manual labels for one intentionally vulnerable application. Section VI candidly lists limitations such as the small corpus, the presence of false negatives, and the lack of scalability studies, but the conclusion still claims 'significant' mitigation and 'enhanced reliability and accuracy.' The absence of a SAST/DAST baseline, of ablations, and of released prompts and configuration details makes the empirical claims non-reproducible and not testable from this manuscript.","section":"Section IV-C, Section VI"}],"minor_comments":[{"comment":"The sentence 'The accuracy for the *original 8 Vuldroid vulnerabilities* dropped to 75' appears to be cut off; state the percentage explicitly and define how accuracy is computed.","section":"Section IV-B"},{"comment":"The table caption and text call the output 'Confirmed (True) / Rejected (False) by MoA,' but the note admits an author adjustment; label the columns as 'Reported final judgment' and clearly indicate which entries are raw model outputs and which are annotations.","section":"Section IV-C, Table V"},{"comment":"The GitHub repository link is mentioned but its contents are not described; if code is to be released, provide a reproducibility appendix with prompts, model versions, API configurations, and the RAG knowledge-base construction details.","section":"Section III"},{"comment":"Table IV labels 'Insecure Input Validation' as a true positive, although Vuldroid's documented vulnerability list does not explicitly include it; specify the ground-truth basis for this annotation.","section":"Section IV-B, Table IV"}],"recommendation":"reject","confidential_remarks":"I recommend rejection. The paper reads as a preliminary engineering report rather than a validated research contribution: the only comparison that could support the central claim is confounded by simultaneous changes in model, prompt, and scoring procedure, and one of the reported results was adjusted on plausibility. I would encourage the authors to conduct a controlled evaluation with ablation arms, release their prompts and pipeline scripts, and report standard detection metrics before resubmitting."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, the paper is not a waste of time: the idea of using a mixture of agents to verify LLM vulnerability reports is a sensible way to attack the hallucination problem, and the workflow described in Section III is easy to follow. Second, the evaluation does not establish that the idea works. The one comparative result, Experiment 3, changes the model, the prompt, and the retrieval context all at once, so you cannot attribute the improvement to RAG+MoA. The author actually admits relabeling 'Hardcoded Credentials' from False to True on plausibility. That alone sinks the claim.\n\nWhat is genuinely new: combining RAG and MoA for this task, with MoA used as a verifier rather than a generator. The paper also gives an honest account of LLM limitations and says outright that fine-tuning is out of scope. The limitations section is candid about false negatives, cost, and scalability, which is more than many papers do.\n\nWhere it falls down: no quantitative metrics, no error bars, no ablations, no code or prompts released, one intentionally small app, and the vulnerability list is taken straight from the app's documentation. The internal inconsistency in Experiment 1 (4/8 vs 7/8) is minor compared to the Experiment 3 confound, but it does not help. The conclusion that the methodology 'significantly mitigated the false positive problem' is an overclaim; the evidence shows one hallucinated candidate rejected in a small manual exercise.\n\nThe citation pattern is fine. RAG and MoA are both cited to the original sources, and LLBezpeky is cited as prior LLM work. No obvious gaps.\n\nWho should read it: anyone designing an LLM-based scanner might borrow the verification idea, and the workflow description is a reasonable starting point. As a research paper, it needs a real evaluation: run RAG-only, MoA-only, and the full pipeline on a standard benchmark, with the same models and prompts, and publish the prompts and raw outputs. Without that, the central claim is unsupported.\n\nMy position: I would not publish this as is, and I would not spend referee effort on it yet. If the author comes back with a proper ablation and artifacts, the idea deserves a hearing. For now, it is a useful engineering note, not a result.","headline":"Plausible RAG+MoA recipe, but the one experiment that matters changes three variables at once and relabels a row; the false-positive claim is not supported.","tokens_in":11090,"tokens_out":2472,"would_cite":false,"duration_ms":24847,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"RAG plus collaborative LLM agents cut false positives in code audits","keywords":["large language models","vulnerability detection","retrieval-augmented generation","mixture-of-agents","false positives","Android security","prompt engineering"],"falsifier":"An ablation study on the same vulnerable Android app: run the identical candidate list through four conditions — single closed-source model with no RAG, the same model with RAG only, open-source MoA without RAG, and open-source MoA with RAG — and count how many times the spurious \"Insecure Design\" candidate is rejected. If RAG and MoA together do not reject it more often than the single model does, the paper's central claim is unsupported. Alternatively, if a single model with just RAG already rejects it, then MoA adds nothing.","tokens_in":9952,"feed_emoji":"🛡️","tokens_out":4443,"duration_ms":35063,"temperature":0.7,"pith_summary":"The paper argues that asking a single large language model to scan source code for vulnerabilities produces too many false alarms, but combining retrieval-augmented generation with a mixture-of-agents architecture can ground the analysis in current vulnerability knowledge and let several models cross-check each other's verdicts. On a deliberately vulnerable Android application, the author shows that a basic single-model prompt correctly identified several known flaws but also invented at least one vague finding (\"Insecure Design\"). Feeding the same candidates through a RAG-fed sequence of open-source models rejected that spurious finding while confirming the true positives, which the paper takes as evidence that RAG+MoA meaningfully lowers false positive rates without sacrificing detection coverage. The broader aim is a dependable, low-cost LLM-based alternative to traditional static and dynamic analysis tools.","feed_headline":"RAG plus collaborative LLM agents cut false positives in code audits","feed_subtitle":"Grounding scans in current knowledge and letting multiple models verify findings suppresses fabricated alerts.","key_machinery":"The central mechanism is a two-part pipeline: (1) Retrieval-Augmented Generation (RAG), which queries a vector database of vulnerability descriptions, code examples, and remediation advice and slots the retrieved text into the prompt as an open-book context; and (2) a Mixture-of-Agents (MoA) architecture, in which each LLM agent receives the source snippet, the vulnerability name, the RAG context, and the previous agent's assessment, iteratively refining and cross-checking the verdict. The RAG half counteracts knowledge cutoffs and context-window limits; the MoA half counteracts hallucination by making the final conclusion the product of several independent models' agreement.","core_discovery":"In the paper's own terms, the central discovery is that grounding each vulnerability check in retrieved external knowledge and then running the check through a chain of cooperating LLMs filters out hallucinated findings that a single-model pass would report. The evidence is the third experiment, in which candidates flagged by an earlier single-model scan were re-examined by a RAG+MoA pipeline built from open-source models; the pipeline confirmed the true positives and rejected the single questionable \"Insecure Design\" claim. The author contends this makes LLM-based detection reliable enough to be practical, and shows open-source models can do the verification work.","pith_inferences":["A key test implied but not run is an ablation: single model with RAG, and MoA without RAG, on the same candidate list; if rejecting the false positive requires both components, that would pin the mechanism and rule out the model-swap confound.","The paper's own numbers suggest that the main remaining bottleneck is missed vulnerabilities, not false alarms; a practical deployment would need to iterate the vulnerability candidate list rather than verify only whatever a first pass surfaces.","If the false-positive suppression holds on industrial codebases, the architecture could slot into CI/CD pipelines as a triage stage that prioritizes which warnings a human auditor must read, rather than as a final judge."],"forward_implications":["Applying RAG+MoA to a broader vulnerability list and to the whole codebase rather than just re-checking earlier findings could also reduce false negatives, the two known misses in the paper's experiments.","Because the MoA stage used only open-source models, the approach promises cost-effective and customizable security analysis without reliance on closed APIs.","The pipeline's intermediate agent outputs provide partial interpretability: a reviewer can see which agent challenged or confirmed a finding, unlike a single black-box model.","The same grounding-and-verification pattern should transfer to other languages and to non-Android contexts, though prompts and knowledge base need retuning."],"supporting_citations":[{"why":"Supplies the Mixture-of-Agents architecture and its benchmark results that motivate collaborative verification.","marker":"[11]"},{"why":"Supplies the Retrieval-Augmented Generation method for grounding LLM outputs in external knowledge.","marker":"[12]"},{"why":"Establishes the hallucination problem that the paper's RAG+MoA approach is designed to mitigate.","marker":"[10]"},{"why":"Prior work on LLM-based vulnerability detection that the paper builds on and compares against.","marker":"[13]"},{"why":"Characterizes the high false positive rates of traditional SAST/DAST tools that motivate the need for a new approach.","marker":"[2]"},{"why":"Shows the limitations of minimalistic LLM prompting for security evaluation, which the paper extends.","marker":"[8]"}],"fun_headline_variants":["RAG and multi-agent review slash false positives in LLM code audits","Grounding vuln scans in retrieved knowledge filters hallucinated alerts","Open-source LLM agents verify findings, cutting fabricated vulnerability reports","Agent collaboration and retrieval reduce unreliable LLM vulnerability detection"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire conclusion rests on the premise that the improvement in Experiment 3 was caused by the RAG and MoA components, when the experiment also changed the underlying models and re-scored the candidates differently than Experiment 2 did.","fun_headline_variants_meta":{"raw":{"variants":["RAG and multi-agent review slash false positives in LLM code audits","Grounding vuln scans in retrieved knowledge filters hallucinated alerts","Open-source LLM agents verify findings, cutting fabricated vulnerability reports","Agent collaboration and retrieval reduce unreliable LLM vulnerability detection"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.0003,"raw_usage":{"total_tokens":1688,"prompt_tokens":857,"completion_tokens":831,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":473,"completion_tokens_details":{"reasoning_tokens":760}},"tokens_in":473,"tokens_out":831,"duration_ms":7475,"temperature":1.0,"reasoning_tokens":760,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:16:59.489086+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"An ablation study on the same vulnerable Android app: run the identical candidate list through four conditions — single closed-source model with no RAG, the same model with RAG only, open-source MoA without RAG, and open-source MoA with RAG — and count how many times the spurious \"Insecure Design\" candidate is rejected. If RAG and MoA together do not reject it more often than the single model does, the paper's central claim is unsupported. Alternatively, if a single model with just RAG already rejects it, then MoA adds nothing.","supporting_citations":[{"cited_title":"Mixture- of-agents enhances large language model capabilities,","cited_arxiv_id":null,"evidence_quote":"Supplies the Mixture-of-Agents architecture and its benchmark results that motivate collaborative verification."},{"cited_title":"Retrieval-augmented generation for knowledge-intensive NLP tasks,","cited_arxiv_id":null,"evidence_quote":"Supplies the Retrieval-Augmented Generation method for grounding LLM outputs in external knowledge."},{"cited_title":"MOB-Net: Limb-modularized Uncertainty Torque Learning of Humanoids for Sensorless External Torque Estimation","cited_arxiv_id":"2402.11221","evidence_quote":"Prior work on LLM-based vulnerability detection that the paper builds on and compares against."},{"cited_title":"Mitigating program security vulnera- bilities: Approaches and challenges,","cited_arxiv_id":null,"evidence_quote":"Characterizes the high false positive rates of traditional SAST/DAST tools that motivate the need for a new approach."}],"review_version":1}