{"id":"14b43f18-3f56-404c-978b-d6572bba90d3","arxiv_id":"2412.08014","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"MAGIC uses three collaborating LLM agents to generate scene-aware adversarial patches and place them in real-world images, achieving higher attack success against YOLO and DETR detectors than the natural diffusion attack baseline.","lead":"This paper introduces MAGIC, a system of three AI agents that generate and place deceptive patches in images of driving scenes to fool object detectors into seeing stop signs that are not there. A smart generalist might read it because it demonstrates how large language models can automate physical-world attacks on autonomous driving perception, and highlights the ease of creating such attacks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Physical evidence supports a proof of concept, not the stated driving-scenario claim; static scenes and deliberately unobserved patches leave D2P transfer under real driving conditions untested.","rationale":"The reader's weakest assumption correctly identifies the physical-transfer gap as the load-bearing point. The strongest claim is that MAGIC is effective in real-world driving scenarios, but the physical validation is confined to two static scenes under standard conditions, with the patches deliberately not seen by vehicles. The paper's own limitations section confirms this scope, so the concern is about overclaiming rather than internal fraud or error. The digital comparative results, ablations, and the LLM-agent framework are genuine contributions, and the paper is transparent about the proof-of-concept nature of the physical tests. Because the evidence supports a conditional accept but not a full accept of the driving-scenario claim, the reader's CONDITIONAL verdict is appropriate. A controlled moving-vehicle test would settle the remaining question; if the frame-level ASR in real traffic is comparable to the static-scene results, the concern is resolved, and if it drops to chance, the headline physical claim should be weakened. No change to the reader's verdict is needed.","tokens_in":21004,"tokens_out":8752,"duration_ms":96940,"concrete_test":"Mount a camera on a vehicle (or use a roadside camera with actual passing traffic) and drive past the printed patch at 10-40 km/h under at least two lighting/weather conditions; record video, run YOLOv5, RT-DETR, and YOLOv10 on every frame, and report frame-level ASR with a 95% confidence interval plus a no-patch false-positive rate. If frame-level ASR at the 0.5 threshold is not significantly above the no-patch baseline, the physical-transfer claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that MAGIC 'can effectively generate context-aware patch, deploy the patch in real world and attack widely applied object detectors' in driving scenarios. The digital experiments are substantial, but the physical evidence is limited to two static scenes (a bus stop bay and a road next to a college), and the authors state that 'we ensured the patches were not observed by vehicles on the road.' The reported physical results are single captures at fixed placement, with no frame-level ASR, no multiple trials, no moving-vehicle camera, and no comparison against a no-patch false-positive baseline. The Limitations section explicitly concedes that 'the current environments primarily serve as proofs of concept under standard conditions, rather than encompassing a broader range of factors such as varying weathers, daylight and nighttime conditions, or noisy camera inputs.' The central claim therefore rests on an untested bridge: that a static, deliberately unobserved patch with favorable lighting and camera geometry transfers to an actual ego-vehicle detector under motion, changing illumination, and natural camera noise. This is not an internal inconsistency, but it is a real overreach from the presented evidence.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MAGIC, a framework that uses three collaborative multimodal LLM agents (GAgent, DAgent, EAgent) to generate physical adversarial patches with text-to-image models and automatically determine their deployment location and scale in a given scene. The authors reformulate physical adversarial attacks as a one-shot patch generation problem and validate the method digitally on six nuImage environments against YOLOv5, RT-DETR, and YOLOv10, reporting attack success rates (ASR) of 66–92% and consistent improvements over NDDA baselines. They also present physical experiments in a few static real-world scenes and claim that MAGIC can effectively generate context-aware patches, deploy them in the real world, and attack widely used object detectors.","tokens_in":21228,"tokens_out":6187,"duration_ms":59503,"significance":"If the claims hold, the paper makes a useful contribution by showing that LLM agents can orchestrate scene-aware adversarial patch generation and deployment, a task that previous diffusion-based attacks did not address. The digital experiments are substantial: six environments, three detectors, and ablations that support the role of each agent. The analysis in Section 3 of how scene context and deployment affect patch effectiveness is a valuable empirical addition. However, the physical validation is thin and the naturalness evaluation is potentially circular, so the current evidence supports a proof of concept rather than the stronger driving-scenario claim in the abstract and conclusion. The framework's modular design and the explicit study of deployment factors are strengths that make the central idea worth further development.","major_comments":[{"comment":"There is an internal inconsistency between the EAgent's acceptance criterion and the reported ASR values. The EAgent is described as iterating until ci = AND(D(Ii) ≥ θ, E(Ii,γ) ≥ δ) is true, meaning every output patch should satisfy D(Ii) ≥ θ. Yet Table 1 reports MAGIC ASR values of 66–92% at a confidence threshold of 0.5, and Table 2 reports 46–74% at 0.8. If θ is the detection confidence threshold, ASR at that threshold should be near 100% for the final patches. Please clarify whether the refinement loop has a maximum iteration that can return non-conforming patches, whether the ASR is computed over intermediate patch proposals, or whether the θ used by the EAgent differs from the evaluation threshold. As written, the quantitative central claim is difficult to interpret.","section":"§4.4 and §5.2"},{"comment":"The physical experiments are limited to two static scenes in the main paper (a bus stop bay and a road next to a college), and the authors state that the patches were not observed by vehicles on the road. The conclusion nevertheless claims that MAGIC \"can effectively generate context-aware patch, deploy the patch in real world and attack widely applied object detectors.\" The Limitations section (Section 6) explicitly concedes that the environments \"primarily serve as proofs of concept under standard conditions,\" with no varying weather, daylight/nighttime, or noisy camera inputs. The physical evidence therefore supports only a proof-of-concept claim, not the full driving-scenario claim. Please either temper the abstract/conclusion to match the evidence or add physical experiments with a moving camera, varied conditions, and multiple trials to support the stronger claim.","section":"§5.3 and Conclusion"},{"comment":"The naturalness scores used to validate the \"visual naturality\" contribution are computed by an LLM (GPT-4o) that belongs to the same family as the EAgent that makes the acceptance decision, and no human perceptual study is reported. Since the EAgent uses this LLM to decide whether a patch is natural, the reported naturalness scores in Supplementary Table 4 are not an independent measure of stealthiness. This circularity weakens the claim that MAGIC produces contextually natural patches. A human evaluation or at least an independent judge model would be needed to support the naturalness claim.","section":"§4.4 and Supplementary Table 4"},{"comment":"The digital evaluation uses only six environment images (one per camera view from nuImage), and the ASR results are reported as point estimates without error bars, confidence intervals, or significance tests. Given that the consistent ASR improvement over baselines is the main quantitative evidence for the method's effectiveness, the small number of environments and the absence of variance reporting make it hard to assess whether the improvements are statistically reliable. Please report standard deviations over patch samples or multiple runs, or otherwise justify the stability of the reported ASR values.","section":"§5.1 and Table 1"}],"minor_comments":[{"comment":"The table header contains a typo: \"Bblation\" should be \"Ablation\".","section":"Table 3"},{"comment":"The subsection heading \"Deployment Naturality\" uses an uncommon term; \"Deployment Naturalness\" would be consistent with the rest of the paper.","section":"§5.2"},{"comment":"The values of the thresholds θ and δ are not given in the main text, while the supplementary material defines CONF_THRESHOLD as 0.80 in the prompt. The main experiments evaluate ASR at both 0.5 and 0.8. Please clarify which threshold the EAgent uses and how it relates to the evaluation thresholds.","section":"§4.4 and Supplementary §3"},{"comment":"The paper calls the task a \"one-shot patch generation problem\" but the pipeline is iterative (EAgent refines the prompt across rounds). Please clarify the distinction or adjust the terminology.","section":"§4.1 and §4.5"}],"recommendation":"major_revision","confidential_remarks":"The core idea of using LLM agents for context-aware adversarial patch generation is timely and the digital results are promising. The main concerns are the unexplained gap between the EAgent acceptance rule and the reported ASR, the circularity of the LLM-based naturalness evaluation, and the overstatement of the physical evidence in the conclusion. These are fixable with clarifications and a more measured framing, so I do not recommend rejection. I would also encourage the authors to include error bars on the ASR numbers and to consider a small human study for naturalness, as those would substantially strengthen the paper."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a serious referee, but the paper is better than its abstract. The genuinely new piece is the integration: three LLM agents (generation, deployment, self-examination) wrapped around a diffusion model, plus the empirical demonstration in Sec. 3 that NDDA patches degrade sharply after environmental deployment and behave differently across scenes. That is a real, useful observation about the prior work. The digital experiments are the core: across six nuImage environments and YOLOv5, RT-DETR, YOLOv10, MAGIC's ASR clearly beats the NDDA baselines, and the ablations support the contribution of each agent. The ASR numbers are measured against external detectors, so the headline results are not circular.\n\nNow the soft spots. The physical validation is thin. Two static scenes, patches deliberately not observed by road vehicles, single captures, no frame-level ASR, no no-patch false-positive baseline. The authors acknowledge in the Limitations that these are proofs of concept under standard conditions with no weather, lighting, or camera noise variation. That is honest, but it means the conclusion \"deploy the patch in real world and attack widely applied object detectors\" overreaches: the evidence supports a controlled proof of concept, not an attack on driving perception under real traffic. The naturalness scores are produced by an LLM from the same family as the EAgent that decides acceptance; not a fatal flaw but worth noting, since naturalness is a key claimed benefit. The ASR numbers come without error bars or significance tests, and no code is released. The free parameters (theta, delta, SoM slider) are standard for such a pipeline and not a concern.\n\nOverall, the digital evidence and the framework are a solid contribution for the adversarial-ML community. The physical claims need either much more data or a more modest framing. I would send this to peer review with a request to focus the revision on closing the gap between the driving-scenario language and the actual experiments.","headline":"Digital results are solid and the agent framework is a genuine novelty; the physical validation is too thin to support the driving-scenario claims.","tokens_in":21715,"tokens_out":2348,"would_cite":true,"duration_ms":23341,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that physical adversarial patches become far more effective when three LLM agents jointly generate, place, and refine them for a given scene, achieving 66 to 92 percent attack success rates across six digital environments…","keywords":["physical adversarial attack","adversarial patch","LLM agents","object detection","autonomous driving","text-to-image generation","scene context","prompt engineering"],"falsifier":"A live-traffic or dynamic-scene physical test in which a printed MAGIC patch is placed where an approaching vehicle can see it, under changing weather, lighting, and camera noise; if the detector's confidence on the patch falls below the 0.5 success threshold (or the patch is no longer recognized as a stop sign), the physical-world effectiveness claim is contradicted.","tokens_in":20815,"feed_emoji":"🛑","tokens_out":6993,"duration_ms":63344,"temperature":0.7,"pith_summary":"This paper is trying to establish that physical adversarial patch attacks against object detectors should be treated as a one-shot, context-aware generation problem rather than a standalone patch-optimization problem. It claims that scene context and patch placement largely determine whether a printed patch will fool a detector, and that a pipeline of three collaborating multimodal LLM agents can handle both generation and deployment automatically. The authors report that their framework, MAGIC, reaches average attack success rates of 80.66 to 90.66 percent across six nuImage environments when attacking YOLOv5, RT-DETR, and YOLOv10, far above the diffusion-only baseline, and that printed patches remain effective in real-world scenes. A sympathetic reader would care because this turns adversarial patch creation into a high-level, language-driven process that could be used to stress-test perception systems in autonomous driving.","feed_headline":"LLM-chosen patch placement fools YOLO and DETR at 66–92%","feed_subtitle":"Context-aware adversarial patches from a three-agent pipeline beat diffusion-only attacks on driving-scene detectors.","key_machinery":"The load-bearing mechanism is a three-agent collaborative loop: GAgent converts the attack objective into a deceptive text prompt for Stable Diffusion v2; DAgent uses set-of-mark visual prompting to read the scene and propose a natural placement and scale for the resulting patch; EAgent inspects the deployed image through both the target detector and a naturalness-scoring LLM and, if either threshold fails, rewrites the prompt for the next iteration. The loop embodies the paper's reformulation: one-shot physical patch generation is framed as iterated prompt engineering guided by scene-aware deployment and self-critique.","core_discovery":"On its own terms, the paper's central discovery is that the effectiveness and stealthiness of a diffusion-generated adversarial patch are not intrinsic to the patch: both depend heavily on the environment in which it is deployed and on where and at what scale it is placed. MAGIC operationalizes this by having a generation agent craft a text prompt for a text-to-image model, a deployment agent select a contextually plausible location and scale using set-of-mark prompting, and a self-examination agent score both detector confidence and human-perceived naturalness before deciding whether to iterate. The reported result is that this collaborative loop produces patches that are detected as stop signs with high confidence by YOLOv5, RT-DETR, and YOLOv10 while looking like abstract art to human observers, with digital attack success rates of 66 to 92 percent depending on environment and detector and successful physical attacks in two real scenes.","pith_inferences":["Beyond the paper: because placement rather than patch texture alone drives success, the same critic-loop design could be repurposed to probe other perception tasks (segmentation, depth, tracking) by swapping the detector for the target model and keeping the naturalness judge.","Beyond the paper: a defensive counterpart, running MAGIC in a sandbox to anticipate where a threat actor would place a patch, would be a natural next step, but the paper only lists OCR checks and robustified training as possible mitigations, not a developed defense.","Beyond the paper: the framework's dependence on a particular commercial multimodal LLM means the reported numbers may shift if that model is updated; testing across LLM backends would reveal how much of the effectiveness comes from the agent loop rather than the underlying language model."],"forward_implications":["If the central claim holds, printed adversarial patches can be produced for a given scene without any gradient-based optimization, using only a text prompt and an image of the environment.","The approach should transfer to other traffic-sign categories and to other detection architectures, because the generation and placement decisions are expressed in natural language rather than tied to a specific model's gradients.","The 0.8-threshold results (average 58.66 percent attack success rate) imply that even confidence-gated detection pipelines remain partially vulnerable to context-aware patches.","High naturalness scores (0.89 to 0.99 across the six digital environments) mean the patches are likely to go unnoticed by human observers, increasing the practical risk to perception systems."],"supporting_citations":[{"why":"Natural Denoising Diffusion Attack is the baseline this work extends; its prompt-based diffusion patches are re-tested before and after deployment.","marker":"[47]"},{"why":"Stable Diffusion v2 is the text-to-image generator used by both GAgent and the NDDA baseline.","marker":"[45]"},{"why":"Set-of-mark prompting supplies DAgent's scene-grounded visual grounding for choosing patch location and scale.","marker":"[68]"},{"why":"Supplies the non-robust-features explanation for why prompts that remove robust features yield adversarial yet natural-looking patches.","marker":"[32]"},{"why":"nuImage environment images used for digital evaluation come from this dataset.","marker":"[17]"},{"why":"YOLOv5 is one of the three target detectors used to measure attack success rate.","marker":"[35]"},{"why":"RT-DETR is the third detector used to show the attack transfers across architecture families.","marker":"[40]"},{"why":"YOLOv10 serves as the more robust detector and the main evaluation target.","marker":"[54]"}],"fun_headline_variants":["LLM agents choose patch spot to fool YOLO and DETR up to 92%","Context-aware adversarial patches: LLM trio tricks detectors 66-92%","Multi-agent LLM places patches that fool YOLO and DETR up to 92%","Patch placement by LLM agents boosts attack success to 92% on YOLO/DETR"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The physical demonstrations were run in two static scenes and the patches were deliberately kept out of view of moving vehicles, so the central claim assumes that success under these controlled conditions carries over to real driving conditions with moving traffic, varying weather, lighting, and camera noise.","fun_headline_variants_meta":{"raw":{"variants":["LLM agents choose patch spot to fool YOLO and DETR up to 92%","Context-aware adversarial patches: LLM trio tricks detectors 66-92%","Multi-agent LLM places patches that fool YOLO and DETR up to 92%","Patch placement by LLM agents boosts attack success to 92% on YOLO/DETR"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000535,"raw_usage":{"total_tokens":2602,"prompt_tokens":1003,"completion_tokens":1599,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":619,"completion_tokens_details":{"reasoning_tokens":1501}},"tokens_in":619,"tokens_out":1599,"duration_ms":10895,"temperature":1.0,"reasoning_tokens":1501,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T18:18:29.535069+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A live-traffic or dynamic-scene physical test in which a printed MAGIC patch is placed where an approaching vehicle can see it, under changing weather, lighting, and camera noise; if the detector's confidence on the patch falls below the 0.5 success threshold (or the patch is no longer recognized as a stop sign), the physical-world effectiveness claim is contradicted.","supporting_citations":[{"cited_title":"Intriguing properties of diffusion models: An empirical study of the natural attack capability in text- to-image generative models","cited_arxiv_id":null,"evidence_quote":"Natural Denoising Diffusion Attack is the baseline this work extends; its prompt-based diffusion patches are re-tested before and after deployment."},{"cited_title":"High-resolution image synthesis with latent diffusion models","cited_arxiv_id":null,"evidence_quote":"Stable Diffusion v2 is the text-to-image generator used by both GAgent and the NDDA baseline."},{"cited_title":"Set-of-mark prompting unleashes ex- traordinary visual grounding in gpt-4v, 2023","cited_arxiv_id":null,"evidence_quote":"Set-of-mark prompting supplies DAgent's scene-grounded visual grounding for choosing patch location and scale."},{"cited_title":"Adversarial examples are not bugs, they are features","cited_arxiv_id":null,"evidence_quote":"Supplies the non-robust-features explanation for why prompts that remove robust features yield adversarial yet natural-looking patches."},{"cited_title":"Ultralytics yolov5, 2020","cited_arxiv_id":null,"evidence_quote":"YOLOv5 is one of the three target detectors used to measure attack success rate."},{"cited_title":"Detrs beat yolos on real-time object detection, 2023","cited_arxiv_id":null,"evidence_quote":"RT-DETR is the third detector used to show the attack transfers across architecture families."}],"review_version":1}