{"id":"44d90739-ca4c-4f4f-ba91-00a740a434fc","arxiv_id":"2608.01639","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"AutoBypass, a knowledge-grounded multi-agent framework, automatically generates evasive shellcode loaders that bypass all seven tested commercial endpoint protection products, with peak evasion of 90% against Windows Defender.","lead":"A multi-agent AI system that uses a curated knowledge base of known attack techniques can automatically generate and refine malware samples that evade several commercial antivirus and endpoint detection products, with up to 90% success against Windows Defender. The result matters because it shows public security knowledge can be operationalized into automated, continuous EDR resilience testing.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline 90%/86.7% evasion rates measure \"not blocked for 30 s of C2 heartbeat,\" but the paper's own one-month retest shows those rates decay, so the success metric overstates durable EDR evasion.","rationale":"I first identified what must be true for the central claim to hold: AutoBypass must evade the complete detection-and-response loop of each commercial product, not merely avoid an immediate block. The paper's own protocol, however, operationalizes evasion as 30 seconds of C2 heartbeat with no observable process termination, and the one-month retest in Table 8 shows that many samples that passed this bar were subsequently detected. This is a load-bearing concern because every headline percentage in the abstract, RQ1, and RQ3 is computed with that short window. I considered alternative concerns: the unfair baseline in Table 7, the absence of confidence intervals, and the heuristic alert inference in Table 5. Those are real but secondary. Even if the baseline comparison were perfectly fair and the alert inference were perfect, a 30-second success criterion would still overstate durable resilience against cloud telemetry and retrospective detection. I credit the paper for reporting the decay explicitly and for acknowledging in Section 7 that alert attribution is heuristic; this candor makes the issue addressable rather than fatal. The concrete test I propose would settle the question by measuring the same pipeline under a longer, console-verified success criterion. If the 24-hour rates match the 30-second rates, the concern does not land. If they drop materially, the central quantitative claims should be re-scaled and the paper's contribution reframed as measuring short-window bypass rather than sustained endpoint evasion. The reader's verdict of CONDITIONAL remains appropriate, so I recommend no change.","tokens_in":23349,"tokens_out":4182,"duration_ms":41542,"concrete_test":"Re-run the RQ1 protocol on the Windows Defender and Trend Micro cells (N=30 per payload) with a stricter success criterion: success requires the process to remain alive and the C2 heartbeat to persist for at least 24 hours, with periodic queries of the EDR management console (or Windows Event Logs and vendor telemetry) confirming that no alert was raised for that endpoint during the window. Compare these 24-hour evasion rates against the 30-second rates in Table 6. If the 24-hour rates fall by more than about 15 percentage points, the 30-second metric is not measuring evasion against the full detection loop, and the headline claims would need to be re-scaled.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that AutoBypass \"bypassed\" all seven targets rests on a success criterion defined in §4.3.3: the payload establishes a C2 connection and maintains heartbeat for more than 30 seconds without being blocked. This is not equivalent to \"without triggering EDR alerts\" as stated in §5.1, because the Tester monitors OS-level phenomena (process survival, file deletion, network connectivity) and does not query the EDR management console for alerts that are logged but not immediately acted upon. A detection that triggers cloud retro-analysis after the 30-second window is counted as evasion. The paper's Table 8 provides direct evidence that the window is too short: the same protocol's rates drop from 87.8% to 76.7% for Windows Defender, 85.6% to 73.3% for Trend Micro, and 50.0% to 37.8% for Bitdefender after one month, with the authors attributing the decay to vendor-cloud analysis and new signature generation. Since the contribution is framed as automated EDR resilience assessment, the quantity being measured should be survival under the defender's full detection-and-response loop, including retrospective detection, not 30 seconds of heartbeat. All downstream claims (RQ1 peak rates, RQ3 small-model parity, and the statement \"bypassed every target\") inherit this metric, so the weakest load-bearing premise is the definition of evasion itself.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents AutoBypass, a multi-agent LLM framework for automated EDR evasion assessment. A knowledge base is built from public blogs, PoC repositories, detection-rule databases, and OPSEC documents; a Strategist agent selects evasion techniques, a Coder synthesizes polymorphic C++ shellcode loaders, a Builder compiles and pre-checks binaries, and a Tester executes samples against live commercial EDRs, infers detection causes from OS telemetry, and feeds failures back into the knowledge base. The evaluation against seven commercial endpoint products reports peak evasion rates of 90% against Windows Defender and 86.7% against Trend Micro, claims that all seven targets were bypassed, and uses ablations to argue that the knowledge base raises 8B open-weight models from 27--53% to 43--83% evasion, approaching large proprietary models. The paper also includes a one-month retest showing decay in evasion rates and an explicit acknowledgment that alert reasoning is heuristic and that the full knowledge base is withheld from release.","tokens_in":23458,"tokens_out":5595,"duration_ms":52962,"significance":"If the reported numbers survive a more defensible success definition, this is a useful contribution to automated adversary emulation. The system was actually built and run; the primary metric is measured against external commercial products rather than a self-defined detector; the ablation study is informative; and the one-month retest plus the explicit discussion of alert-inference limits are signs of honest reporting. The paper does not rely on circular reasoning: the knowledge base is built from public sources and used for generation, while success is judged by the target EDRs themselves. The main weaknesses are that the success metric measures only 30 seconds of C2 heartbeat rather than the full detection-and-response loop, that the one-month decay is confounded by the authors' own VirusTotal submissions, that the baseline comparison uses different denominators, and that the released artifact excludes the full knowledge base, so the central claims are not independently reproducible as described.","major_comments":[{"comment":"The success metric is load-bearing and currently inconsistent. Section 4.3.3 defines evasion as establishing a C2 connection and maintaining heartbeat for over 30 seconds without being blocked, while Section 5.1 restates the protocol as executing 'without triggering EDR alerts.' The Tester monitors OS-level phenomena (process survival, file deletion, network connectivity) and does not query the EDR management console for alerts that are logged but not immediately acted upon, so a sample that is flagged afterwards by cloud retro-analysis is counted as a success. Table 8 is direct evidence that the window is too short: the same protocol's rates drop from 87.8% to 76.7% for Windows Defender, 85.6% to 73.3% for Trend Micro, and 50.0% to 37.8% for Bitdefender after one month. Because the headline RQ1, RQ3, and 'bypassed every target' claims all inherit this metric, the authors should report both immediate survival and post-retrospective detection from the EDR consoles at longer horizons (e.g., 24 hours and one week), and re-state the claims accordingly.","section":"4.3.3 and 5.1; Table 8"},{"comment":"The one-month robustness claim is confounded by the authors' own VirusTotal submissions. The paper states that all 630 AutoBypass samples were uploaded to VirusTotal for the broader assessment, and later attributes the one-month decay partly to 'public sandbox submission accelerat[ing] signature generation.' Since the retested samples are the same samples that were already submitted to VirusTotal, the measured decay conflates vendor cloud telemetry with the effect of the authors' own VT exposure. To support the temporal-robustness interpretation, the authors should retest a held-out set of successful samples that were never uploaded to VirusTotal, or at least quantify the decay separately for VT-exposed and non-exposed samples.","section":"5.3, Figure 6, and Table 8"},{"comment":"The comparison against BOAZ, Inceptor, and Dante-7B is not apples-to-apples. The table note says 'Evasion rates are calculated based on successfully generated samples, not total generation attempts,' while the AutoBypass rates in Table 6 are reported as successes over 30 iterations per cell. If the baseline rates are conditional on successful compilation/generation and the AutoBypass rates are marginal over attempts, the comparison inflates the gap. The authors should report both marginal and condition-on-success rates for every method with explicit denominators.","section":"Table 7"},{"comment":"The alert root-cause inference is acknowledged to be heuristic, yet it drives the closed-loop adaptation that is central to the framework's contribution. The mapping from telemetry phenomena to root cause (static detection, behavioral blocking, network interception, memory scanning) is never validated, and the paper concedes it 'cannot guarantee absolute attribution accuracy.' The authors should validate the inference against a ground-truth source where available, for example Elastic Security's verbose alerts, or against a set of samples whose triggering mechanism is independently known, and report a confusion matrix or agreement rate. Without that, the historical-feedback mechanism and the RQ2 conclusions about component contributions rest on unverified attributions.","section":"Table 5 and Section 7"},{"comment":"The statistical support is insufficient for several of the paper's comparative claims. Each primary cell has N=30, and Table 8 has N=90, but no confidence intervals or significance tests are reported; differences such as 90.0% vs 86.7% vs 83.3% are within plausible sampling error, and zero-success cells (e.g., DeepSeek-V3 and the 8B models against Bitdefender and Elastic) have wide binomial intervals. The claim that 8B models approach or match large proprietary models needs error bars or exact tests (e.g., Fisher's exact test) on the underlying counts, not just point estimates.","section":"Tables 6, 9, and 10"},{"comment":"The Artifacts section explicitly states that withholding the complete pre-populated knowledge base 'reduces the direct reproducibility of our core claims.' Since the KB is the central contribution and the basis for the RQ2/RQ3 conclusions, releasing only three of the least effective techniques makes it impossible for an independent party to reproduce the main experiments. The authors should specify exactly what a third party can verify with the released subset, provide a full schema and example entries for the withheld portion, and state whether the evaluation harness can be run end-to-end on the released subset.","section":"Artifacts"}],"minor_comments":[{"comment":"The x-axis categories '0' and '1-5' are ambiguous, and the text says 'five or fewer detections' while the figure treats zero as a separate bucket; the authors should clarify whether the 62.9% figure includes the zero-detection bucket.","section":"Figure 6"},{"comment":"The knowledge-retrieval quality assessment is based on two experts labeling 691 unique queries, but the paper reports no inter-rater reliability; adding Cohen's kappa or a disagreement count would make the 82.1% hit-rate claim more robust.","section":"5.3"},{"comment":"The limitation section mentions the binary-splitting tool 'VirTest' without a citation or reference; please add a source or remove the name.","section":"Section 7"},{"comment":"The phrase 'Detection-A ware' appears as a line-break artifact in the abstract and introduction; this should be corrected to 'Detection-Aware.'","section":"Abstract and Section 1"},{"comment":"For cells with zero successful evasions, the average token cost is listed as N/A, which is correct, but the corresponding row still reports a success rate of 0.0%; consider adding a footnote that the token statistics are undefined rather than zero for those cells.","section":"Tables 9 and 10"}],"recommendation":"major_revision","confidential_remarks":"The core system appears to have been genuinely built and evaluated against real products, and the authors are transparent about several limitations. The main risk is not fabrication but the definition of the central metric: 30-second heartbeat survival is a much weaker claim than 'bypassed without triggering EDR alerts,' and the one-month decay data underscore this. I would not reject, but the manuscript needs a redefinition or re-scoping of the success metric, a handling of the VirusTotal confound, and a clearer reporting of baseline denominators before the headline claims can be accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Useful paper, and I'd send it out, but with a red pencil on the framing. The core of the work is real: AutoBypass is a multi-agent system that takes public evasion knowledge, structures it into a searchable KB, generates polymorphic shellcode loaders, runs them against seven commercial endpoint products, infers failure causes from telemetry, and iterates. That's actually built and measured, not a proposal. The strongest result isn't the 90% headline; it's the ablation showing the KB lifts 8B open-weight models from 27-53% to 43-83%, bringing them near large proprietary models. That's a concrete, reproducible claim about architecture doing the work, and it matters for the LLM-security community.\n\nThe main soft spot is the success definition. Evasion is 'C2 heartbeat for 30 seconds without being blocked' (§4.3.3). That's a short window, and it doesn't capture retrospective detection via cloud telemetry. The paper's own Table 8 shows one-month retests dropping: Defender 87.8% to 76.7%, Bitdefender 50.0% to 37.8%. So the headline rates are upper bounds on 'not immediately caught,' not durable evasion. The authors acknowledge the decay and attribute it to cloud analysis, which is honest, but the abstract and RQ1 answers still say 'bypassed every target' without that qualifier. That needs fixing.\n\nTwo second-tier issues. First, no confidence intervals or significance tests on N=30; several ablation deltas (e.g., 83.3% vs 60%) would survive, but some one-sample differences won't. Second, Table 7's baseline comparison isn't apples-to-apples: AutoBypass gets up to five refinement attempts, while BOAZ/Inceptor/Dante-7B rates are computed on successfully generated samples only, which deflates the baselines. Also, 'seven EDRs' includes four consumer AV products; the two enterprise EDRs (Elastic, Bitdefender) hold Cobalt Strike to 13-23%, which is a different story than the consumer numbers.\n\nThe circularity concern doesn't land: the central metric is against external commercial products, so the KB is not scoring its own homework. The KB itself is withheld, but they provide a minimal subset for reproducibility; that's a real limitation, acknowledged explicitly.\n\nWho should read this: anyone working on LLM agents for offensive security, automated red teaming, or EDR evaluation. It deserves a serious referee. My recommendation: send to peer review with a request to (a) qualify all headline rates with the 30s/one-month decay caveat, (b) add error bars or more runs, and (c) fix the baseline denominator. The architecture and the KB-amplification finding are solid enough to justify revision rather than rejection.","headline":"A real, built system with a solid KB-amplification result, but the headline 90% evasion rates rest on a 30-second heartbeat window and decay noticeably after one month, so the numbers are encouraging upper bounds, not durable EDR bypass rates.","tokens_in":24125,"tokens_out":2614,"would_cite":true,"duration_ms":22108,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A knowledge-grounded, closed-loop multi-agent framework can turn publicly documented EDR evasion techniques into working payloads, bypassing all seven commercial endpoint security platforms it tested.","keywords":["EDR evasion","multi-agent orchestration","knowledge base","shellcode loader","polymorphic code generation","closed-loop feedback","LLM security","endpoint detection and response"],"falsifier":"Instrument a lab EDR so it logs the exact detecting component for every alert, run the same sample suite on telemetry-connected hosts for 30 days without submitting samples to public sandboxes, and compare the ground-truth detection logs with the framework's inferred root causes and survival times; if a substantial share of samples is flagged only after the first 30 seconds, or the inferred causes disagree with the logs, the 30-second criterion and heuristic inference overstate resilience.","tokens_in":23007,"feed_emoji":"🤖","tokens_out":10119,"duration_ms":82183,"temperature":0.7,"pith_summary":"This paper tries to establish that the knowledge needed to evade modern endpoint detection and response (EDR) products is already public and can be systematically operationalized by an autonomous system: a multi-agent pipeline reads threat intelligence, detection rules, and proof-of-concept code, plans an attack, generates polymorphic payloads, runs them against live defenses, and refines strategy from the alerts. Across seven commercial endpoint security platforms the system bypassed every target, reaching 90% evasion against Windows Defender and 86.7% against Trend Micro AV. The paper further claims that a structured knowledge base is the decisive component, raising the evasion success of small open-weight models from 27–53% up to 43–83%, close to large proprietary models. A sympathetic reader would care because, if true, this turns EDR resilience testing from a manual expert activity into a repeatable automated benchmark, and it exposes a systemic blind spot: payloads that execute inside trusted processes, such as DLL sideloading, outperform standalone executables. The paper's own one-month retest shows evasion rates decaying as cloud telemetry and sandbox submissions generate new signatures, so the headline numbers measure short-horizon evasion rather than long-term stealth.","feed_headline":"Seven commercial endpoint defenses fell to an autonomous AI system","feed_subtitle":"Knowledge base and multi-agent pipeline hit 90% evasion on Windows Defender and 86.7% on Trend Micro.","key_machinery":"The load-bearing mechanism is the Detection-Aware Knowledge Base, a hybrid relational-plus-vector store that turns unstructured threat intelligence, expert analyses, and proof-of-concept code into a structured taxonomy of evasion techniques tagged by loader stage and risk level, OPSEC constraints, and historical failure records; in the implementation it holds 19 evasion techniques, 5 encoding schemes, 68 OPSEC constraints, and 1,537 detection rules. Around it, a multi-agent pipeline with roles for strategy, coding, building, and debugging plans blueprints, synthesizes polymorphic C++ source from verified snippets under stealth constraints, compiles, and self-repairs, while a Tester agent executes samples against live EDRs and uses alert-timing heuristics to infer root causes that update the history. The knowledge base carries the argument because it prevents functional hallucination by grounding code in verified snippets and enables semantic-level pivoting between tradecrafts when one approach is blocked.","core_discovery":"On its own terms, the paper's central claim is that EDR bypass can be modeled as a closed-loop orchestration problem rather than a one-shot payload-generation trick. The system divides the shellcode-loader lifecycle into six stages, selects and composes evasion primitives for each stage from its knowledge base, has agents generate polymorphic C++ and compile it, then infers detection root causes from OS-observable telemetry such as file deletion, process termination, network failure, and delayed termination, and feeds those inferences back into the planner. In evaluation it bypassed all seven targets, with peak success of 90% against Windows Defender and 86.7% against Trend Micro AV, and 62.9% of generated samples triggered five or fewer detections in an aggregate antivirus scan. The authors are explicit that the contribution is not the discovery of new evasion primitives but the autonomous, knowledge-grounded orchestration of known techniques, with the knowledge base lifting 8B open-weight models from 27–53% to 43–83% success.","pith_inferences":["Our inference: the 30-second survival criterion likely understates true resilience; the paper's one-month decay suggests the binding constraint on such attacks is retrospective cloud analysis, so a stronger benchmark would measure survival over days or weeks without sandbox submission.","Our inference: the knowledge-base-as-amplifier result plausibly transfers to other security code-generation tasks where raw LLMs hallucinate, such as detection-rule authoring or exploit scaffolding, where verified snippets and OPSEC constraints would play the same grounding role.","Our inference: the alert-inference heuristics (file deletion within 5 seconds, process death within 30 seconds, termination after 2 minutes) are directly testable; an EDR that logs its own detecting component would let a follow-up study measure attribution accuracy instead of assuming it.","Our inference: because evaluation covers only shellcode loaders, extending the same orchestration to script-based vectors such as PowerShell or WMI could reveal different blind spots and is a natural next experiment."],"forward_implications":["Current commercial EDRs remain vulnerable to polymorphic, knowledge-driven attacks assembled entirely from publicly documented techniques.","A structured knowledge base acts as an intelligence amplifier: computationally efficient 8B models approach large proprietary models' evasion rates, so this kind of assessment no longer requires the most capable LLM.","Techniques that hide in trusted execution contexts, such as DLL sideloading, are a systematic blind spot across tested defenses and deserve vendor attention.","EDR resilience assessment can be automated and run continuously, turning scattered public threat intelligence into repeatable stress tests.","Evasion is temporal: rates decay within a month as cloud telemetry and sandbox submissions generate new signatures, so ongoing mutation, not one-shot generation, is the realistic operating mode."],"supporting_citations":[{"why":"Supplies the six-stage shellcode-loader taxonomy whose stage tags organize the knowledge base's technique entries.","marker":"[19]"},{"why":"Documents user-mode hook bypass via indirect syscalls, a representative manual primitive the framework systematizes into knowledge base entries.","marker":"[35]"},{"why":"Template-based shellcode-loader generator used as a baseline whose deterministic fingerprints the proposed system is claimed to outperform.","marker":"[38]"},{"why":"Template-based evasion framework used as a baseline; its near-zero evasion rates across targets contrast with the proposed system's results.","marker":"[59]"},{"why":"Fine-tuned LLM baseline with a very low compilation success rate, motivating the knowledge base's grounding role in code generation.","marker":"[5]"},{"why":"YARA rule repository parsed into the knowledge base's defensive rules for pre-deployment static scanning.","marker":"[66]"},{"why":"Sigma rule repository parsed into the knowledge base's behavioral detection constraints.","marker":"[55]"},{"why":"Elastic detection-content repository parsed into the knowledge base's defensive rules.","marker":"[23]"},{"why":"Provides the well-known C2 framework payload used as one of the three evaluation workloads.","marker":"[13]"},{"why":"Provides the open-source C2 framework payload used to confirm the framework's generalizability across modern threats.","marker":"[51]"}],"fun_headline_variants":["AutoBypass AI beats all seven EDRs, 90% on Windows Defender","Autonomous knowledge-driven agents slip past every EDR in trial","Closed-loop agent framework evades all 7 commercial defenses","AutoBypass orchestrates known evasion tricks to beat 7 EDRs","Multi-agent system lifts open models to 83% EDR bypass"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that surviving 30 seconds with a stable command-and-control heartbeat, together with the heuristic mapping of alert timing to root causes, is an adequate measure of evasion; the paper's own one-month retest shows rates decaying sharply (for example, Windows Defender from 87.8% to 76.7% and Bitdefender from 50.0% to 37.8%), so if true resilience means avoiding retrospective cloud-based detection for days, the headline rates overstate it.","fun_headline_variants_meta":{"raw":{"variants":["AutoBypass AI beats all seven EDRs, 90% on Windows Defender","Autonomous knowledge-driven agents slip past every EDR in trial","Closed-loop agent framework evades all 7 commercial defenses","AutoBypass orchestrates known evasion tricks to beat 7 EDRs","Multi-agent system lifts open models to 83% EDR bypass"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000636,"raw_usage":{"total_tokens":2949,"prompt_tokens":977,"completion_tokens":1972,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":593,"completion_tokens_details":{"reasoning_tokens":1876}},"tokens_in":593,"tokens_out":1972,"duration_ms":14044,"temperature":1.0,"reasoning_tokens":1876,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T15:06:33.671067+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Instrument a lab EDR so it logs the exact detecting component for every alert, run the same sample suite on telemetry-connected hosts for 30 days without submitting samples to public sandboxes, and compare the ground-truth detection logs with the framework's inferred root causes and survival times; if a substantial share of samples is flagged only after the first 30 seconds, or the inferred causes disagree with the logs, the 30-second criterion and heuristic inference overstate resilience.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the six-stage shellcode-loader taxonomy whose stage tags organize the knowledge base's technique entries."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Template-based shellcode-loader generator used as a baseline whose deterministic fingerprints the proposed system is claimed to outperform."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Template-based evasion framework used as a baseline; its near-zero evasion rates across targets contrast with the proposed system's results."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Fine-tuned LLM baseline with a very low compilation success rate, motivating the knowledge base's grounding role in code generation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"YARA rule repository parsed into the knowledge base's defensive rules for pre-deployment static scanning."},{"cited_title":"In 2020 IEEE symposium on security and privacy (SP)","cited_arxiv_id":null,"evidence_quote":"Sigma rule repository parsed into the knowledge base's behavioral detection constraints."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Elastic detection-content repository parsed into the knowledge base's defensive rules."},{"cited_title":"2020.New Snort, ClamA V coverage strikes back against Cobalt Strike","cited_arxiv_id":null,"evidence_quote":"Provides the well-known C2 framework payload used as one of the three evaluation workloads."},{"cited_title":"2025.AdaptixC2: A New Open-Source Framework Leveraged in Real-World Attacks","cited_arxiv_id":null,"evidence_quote":"Provides the open-source C2 framework payload used to confirm the framework's generalizability across modern threats."}],"review_version":2}