{"id":"ed2a7f36-0d5d-4c1d-8729-7ab762415158","arxiv_id":"2606.02924","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":7.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"ATLAS is the first large-scale benchmark for black-box adversarial attacks on LiDAR perception, revealing that stronger clean-data models are more vulnerable to point injection due to object database sampling augmentations.","lead":"The paper introduces ATLAS, a benchmark that simulates black-box point injection and removal attacks on real LiDAR driving data to test modern perception models. A smart generalist should read it to understand how common training practices can create unexpected safety vulnerabilities in autonomous driving systems.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"Fidelity of simulated point injection/removal to real physical black-box LiDAR attacks is the load-bearing assumption.","rationale":"The reader correctly flagged the simulation-to-reality gap as the weakest assumption; the full text does not add hardware validation that would close it, so the central claim remains conditional on that untested mapping.","tokens_in":1766,"tokens_out":339,"duration_ms":12315,"concrete_test":"Acquire a real Velodyne or Ouster LiDAR, perform controlled physical injection (laser spoofing) and removal (occlusion) on the same scenes used in ATLAS, extract the resulting point clouds, and compute the Earth-mover’s distance between their 3-D intensity histograms and the corresponding ATLAS-simulated clouds; if the median distance exceeds the intra-simulator variance by >2σ, re-run the model evaluations on the hardware point clouds and check whether the asymmetry persists.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The robustness asymmetry and its attribution to object-database sampling both rest on the claim that the ATLAS simulators produce point clouds whose statistical and geometric properties match those inducible by physical attacks on deployed LiDAR hardware. The abstract and method sections describe the simulators as “physically grounded,” yet the paper supplies no direct comparison (e.g., Kolmogorov–Smirnov tests on intensity histograms, angular density, or temporal consistency) between simulated and hardware-induced point sets. If the simulators systematically under- or over-estimate the spatial clustering or intensity profiles that real spoofing or occlusion produce, the observed performance ordering across models could be an artifact of the simulation rather than a property of the perception pipelines.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper introduces ATLAS, a large-scale benchmark simulating physically grounded black-box point injection and removal attacks on LiDAR perception models across real driving sequences. It evaluates a range of state-of-the-art detectors, reports a robustness asymmetry (stronger clean-performance models resist removal attacks better but are more vulnerable to injection), attributes the injection vulnerability to standard object-database sampling augmentations, explores initial mitigations, and releases generation code for reproducibility.","tokens_in":1897,"tokens_out":412,"duration_ms":10309,"significance":"If the simulated attacks faithfully reproduce the statistical and geometric effects of real physical LiDAR spoofing and occlusion, the benchmark would provide the first systematic evidence that current training practices induce architecture-agnostic robustness failures, directly informing safer LiDAR perception development.","major_comments":[{"comment":"Methods (attack simulation): the central robustness-asymmetry and augmentation-attribution claims rest on the unverified assumption that the point-injection and point-removal simulators produce point clouds whose intensity histograms, angular density, and temporal consistency match those inducible by physical black-box attacks on deployed hardware; no Kolmogorov–Smirnov tests, direct hardware comparisons, or ablation on simulator parameters are reported to support this fidelity.","section":"Methods (attack simulation)"},{"comment":"Results (augmentation ablation): the claim that object-database sampling is the root cause of the injection vulnerability is load-bearing yet lacks a controlled experiment isolating this augmentation from other training choices (e.g., intensity scaling, point dropout) while holding architecture fixed; without such isolation the architecture-agnostic conclusion cannot be drawn.","section":"Results (augmentation ablation)"}],"minor_comments":[{"comment":"Abstract and §1: the phrase “physically grounded” is used without a forward reference to the specific physical model or validation metric employed in the simulator.","section":"Abstract and §1"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive comments, which focus on the fidelity of the attack simulators and the strength of the augmentation attribution. We address each point below, indicating planned revisions where appropriate.","responses":[{"response":"The simulators are constructed from established physical models of LiDAR point injection (spoofing) and removal (occlusion) drawn from the existing literature on black-box sensor attacks. The reported robustness asymmetry and its link to training practices are empirical observations under these simulated conditions. We agree that additional validation would strengthen the presentation: the revised manuscript will include an ablation on simulator parameters (injection rate, intensity distribution, angular density) and Kolmogorov–Smirnov comparisons against published real-attack point-cloud statistics where such data exist. Direct hardware replication lies outside the scope of a simulation benchmark and is noted as future work.","revision_made":"partial","referee_comment":"Methods (attack simulation): the central robustness-asymmetry and augmentation-attribution claims rest on the unverified assumption that the point-injection and point-removal simulators produce point clouds whose intensity histograms, angular density, and temporal consistency match those inducible by physical black-box attacks on deployed hardware; no Kolmogorov–Smirnov tests, direct hardware comparisons, or ablation on simulator parameters are reported to support this fidelity."},{"response":"The current manuscript correlates the presence of object-database sampling with elevated injection vulnerability across multiple architectures. To isolate the effect, the revision will add a controlled ablation that holds architecture, optimizer, and all other augmentations fixed while toggling only object-database sampling. This experiment will directly support the architecture-agnostic attribution.","revision_made":"yes","referee_comment":"Results (augmentation ablation): the claim that object-database sampling is the root cause of the injection vulnerability is load-bearing yet lacks a controlled experiment isolating this augmentation from other training choices (e.g., intensity scaling, point dropout) while holding architecture fixed; without such isolation the architecture-agnostic conclusion cannot be drawn."}],"tokens_in":1388,"tokens_out":433,"duration_ms":22314,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The main takeaway is that ATLAS is the first large-scale benchmark that applies black-box point injection and removal attacks to a range of current LiDAR perception models on real driving data. It reports that models which do better on clean benchmarks are more vulnerable to injection attacks than weaker ones, and it traces this to standard object database sampling augmentations.\n\nThe paper does a few things right. It moves past earlier work that focused on attack hardware or older detectors, runs the tests across multiple modern architectures, and releases the generation code so others can reproduce and update the suite. That makes the evaluation extensible and gives the community a concrete starting point for thinking about sensor-level robustness.\n\nThe soft spot is the simulation fidelity. The attacks are described as physically grounded, yet the paper does not show direct statistical comparisons between the generated point clouds and those produced by actual hardware attacks—no checks on intensity distributions, spatial clustering, or temporal patterns. If the simulated injections and removals do not match real sensor behavior, the reported asymmetry and its link to augmentations could be an artifact. That assumption carries the main results, so it is not a minor detail.\n\nThe mitigation directions are presented as initial, which is fair. This paper is aimed at researchers working on LiDAR perception for autonomous driving who want to move beyond clean benchmarks. A reader in that area would find the released code and the scale of the evaluation useful. It deserves a serious referee because the evaluation gap is genuine and the benchmark itself is new, even if reviewers will need to press on the physical realism of the attacks.","headline":"ATLAS gives a useful new benchmark for LiDAR sensor attacks and flags a robustness asymmetry tied to training, but the claims rest on unvalidated simulations of physical attacks.","tokens_in":2365,"tokens_out":394,"would_cite":false,"duration_ms":16583,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"High-performing LiDAR models resist point removal better but prove more vulnerable to point injection attacks.","keywords":["LiDAR perception","adversarial attacks","robustness evaluation","point injection","point removal","autonomous driving","black-box attacks"],"falsifier":"A physical test in which real sensor attacks produce vulnerability patterns that differ from the asymmetry observed in the ATLAS simulations.","tokens_in":2685,"feed_emoji":"🚗","tokens_out":521,"duration_ms":22018,"temperature":0.7,"pith_summary":"The paper introduces the ATLAS benchmark to test LiDAR perception models against simulated black-box sensor attacks that either add or remove points from real driving scans. It shows that models with stronger results on clean benchmarks handle removal attacks more effectively yet suffer greater failures under injection attacks than weaker models. The source of this pattern is traced to standard object database sampling used during training. This matters for autonomous driving because physical attacks on LiDAR can occur without any model access and current evaluation practices overlook these failure modes.","feed_headline":"Stronger LiDAR models fail more on point injection attacks","feed_subtitle":"ATLAS benchmark links common training augmentations to uneven resistance across attack types","key_machinery":"The ATLAS benchmark that generates simulated point injection and point removal attacks across real driving sequences to measure model robustness under black-box conditions.","core_discovery":"Evaluating a broad set of current state-of-the-art LiDAR perception models with ATLAS reveals a robustness asymmetry in which stronger clean-data performers better withstand removal attacks yet are more vulnerable to injection attacks, with the vulnerability traced to object database sampling augmentations in training.","pith_inferences":["Training pipelines may require new augmentation strategies that preserve removal resistance without increasing injection exposure.","Comparable evaluation suites for other sensors such as cameras could reveal whether similar performance-robustness trade-offs exist.","Clean benchmark scores alone may not reliably indicate resistance to physical sensor manipulations in practice."],"forward_implications":["Standard object database sampling during training creates architecture-agnostic failures against injection attacks.","The released generation code enables ongoing reproducible tests as new attack methods appear.","Both attack modes can be studied together to develop mitigations that address the observed asymmetry."],"fun_headline_variants":["ATLAS exposes stronger LiDAR models as more vulnerable to injection attacks","LiDAR clean performance correlates with injection attack failure in ATLAS","Object sampling in training weakens LiDAR models against point injection","ATLAS benchmark finds top LiDAR models fail more on point injection"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"The simulated injection and removal attacks produce effects that match those of real physical black-box attacks on deployed LiDAR sensors.","fun_headline_variants_meta":{"raw":{"variants":["ATLAS exposes stronger LiDAR models as more vulnerable to injection attacks","LiDAR clean performance correlates with injection attack failure in ATLAS","Object sampling in training weakens LiDAR models against point injection","ATLAS benchmark finds top LiDAR models fail more on point injection"]},"model":"grok-4.3","cost_usd":0.005682,"raw_usage":{"total_tokens":2715,"prompt_tokens":669,"num_sources_used":0,"completion_tokens":69,"cost_in_usd_ticks":56824500,"prompt_tokens_details":{"text_tokens":669,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":1977,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":669,"tokens_out":69,"duration_ms":14777,"temperature":1.0,"reasoning_tokens":1977,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-28T14:45:46.149815+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"A physical test in which real sensor attacks produce vulnerability patterns that differ from the asymmetry observed in the ATLAS simulations.","supporting_citations":[],"review_version":1}