REVIEW 3 major objections 8 minor 35 references
Revisiting Physically Realizable Adversarial Object Attack against LiDAR-based Detection: Clarifying Problem Formulation and Experimental Protocols
T0 review · 3 major / 8 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read A device-agnostic framework claims simulation is a valid proxy for physical LiDAR adversarial attacks, validated by transferring a 3D-printed adversarial object to real hardware.
desk verdict A useful standardization effort for LiDAR adversarial object attacks, but the headline sim-to-real claim rests on a single qualitative figure while the paper explicitly excludes real-world testing from its formal protocol. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The engine is the adversarial mesh $M_{adv}$, initialized as a sphere and deformed through learnable vertex displacements $v_i = \tau(v_i^0, \Delta v_i)$ under a box constraint, integrated into LiDAR point clouds via the Möller–Trumbore ray-triangle intersection. Physical feasibility is enforced by Laplacian smoothness $\phi(M_{adv})$ added to the loss with weight $\lambda$, and the attack is driven by Algorithm 1, a gradient-descent loop that in black-box settings updates the mesh only when the target detector's loss decreases. Evaluation rests on standardized attack success rate computed from 3D and BEV mAP at IoU 0.7, plus invisibility metrics (L2 norm, Laplacian, area, volume) over KITTI and CarLA scenes.
What would settle it
Re-run the CarLA transferability experiments with independently annotated 3D boxes for the Car class (e.g., from a different detector or manual labeling) and compare ASR for MR(9) on PointPillar and PointRCNN: if the attack success rate drops materially below the reported 72.80/61.90 (PP) and 46.02/61.90 (PR) values, the ground-truth assumption fails. Alternatively, repeat the physical 3D-print transfer test with a different LiDAR sensor model; if the simulated object no longer hides the car, the simulation-as-proxy claim is hardware-specific.
Extended reading notes
Core claim
The paper's central claim is that the messy, hardware-dependent practice of physical adversarial object attacks reduces to a standardized optimization: minimize a misdetection loss $L(f, G(P, M_{adv}, t))$ over mesh vertices subject to a physical-feasibility predicate, where $G$ is ray-triangle integration, feasibility is enforced by box-constraint reparameterization and Laplacian smoothness, and the loss is chosen from categorized mislocalization/misrecognition/comprehensive designs. It demonstrates across five detectors (PointRCNN, PointPillar, PV-RCNN, Voxel RCNN, SECOND) that this framework reproduces and outperforms the PhyAdv baseline, that misrecognition losses are strongest, that using logits instead of scores does not improve transferability in LiDAR (unlike images), that gradient descent matches or beats Adam, and that larger BEV area of the adversarial mesh consistently improves transferable BEV attack success. The same meshes transfer from KITTI to the CarLA simulator and, in one physical experiment, to a real LiDAR detector.
Load-bearing premise
The CarLA evaluation assumes that 3D bounding boxes generated by the victim detector itself on clean data are valid ground truth; if those predictions are biased, the reported attack success rates and transferability numbers may not reflect true detection failure.
Editorial extensions
If this is right
- Researchers can now compare adversarial object attacks on common ground: the framework fixes mesh initialization, loss taxonomy, optimizer, and ASR/invisibility metrics across KITTI and CarLA.
- Simulation-based evaluation can stand in for physical deployment tests for rooftop-placement attacks, dramatically lowering the cost of robustness assessment.
- Attack design guidance follows: use misrecognition-type losses, prefer gradient descent with per-model step sizes, and enlarge the mesh's BEV footprint to boost cross-model transferability.
- The logit-vs-score result implies that image-derived transferability heuristics do not carry over to LiDAR detectors, pointing to architecture-specific attack mechanisms.
- The physical validation supports using this benchmark for safety compliance exercises, such as assessing whether detectors meet EU AI Act robustness expectations.
Reading between the lines
- The simulation-proxy claim is only directly validated for rooftop objects observed by the top LiDAR lines; whether it extends to ground-level obstacles, occluded placements, or different sensor vertical resolutions remains an open question the paper does not settle.
- Because CarLA ground-truth boxes are generated by the victim detector's own clean predictions, the reported cross-dataset ASR could be inflated if those predictions are biased; an independent label source would be a stronger test of the transferability claim.
- The paper observes that larger BEV area correlates with transferable ASR but does not explicitly optimize for area; adding an area-maximization term to the loss might produce even stronger attacks, a direct testable extension.
- Reflectivity is neutralized in all experiments, so the framework currently isolates geometric attacks; combining intensity-channel perturbations with the geometry optimization could reveal additional physical attack surfaces in real sensors.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a unified, device-agnostic framework for generating and evaluating physically realizable adversarial object attacks against LiDAR-based 3D object detectors. It formalizes the adversarial mesh optimization problem, categorizes misdetection losses into mislocalization, misrecognition, and comprehensive forms, and provides ablations over optimizer, initial mesh, and loss design. The experimental comparison includes white-box and black-box transferability on the KITTI dataset, a cross-dataset evaluation on the KITTI-CARLA (CarLA) dataset, and scenario-based results in the CARLA simulator. The authors also report a small-scale physical experiment with a 3D-printed object and claim that simulation is validated as a proxy for physical feasibility. The paper is positioned as a reproducibility contribution, with open-source code and standardized evaluation protocols.
Significance. If the claims are substantiated, this would be a valuable contribution: the community needs reproducible benchmarks for physically realizable attacks, and the paper provides a structured framework, per-model ablations, and a breakdown of loss formulations. Credit is due for the detailed KITTI experiments, which use external ground truth and are not circular, and for the explicit discussion of reflectivity and constraint limitations. However, the central sim-to-real validation is not supported by the evidence in the manuscript: only a qualitative Figure 1 is offered, and the text explicitly excludes real-world testing from the formal protocol. The CarLA evaluation also relies on ground-truth labels generated by the victim detector itself, which creates a self-referential measurement. The skeptic's concern therefore lands: the headline claim that simulation is an effective proxy for physical feasibility is currently unvalidated, and the dataset-transfer conclusions are weakened. These issues are fixable through additional experiments or a careful reframing of the claims, so the paper warrants a major revision rather than rejection.
major comments (3)
- [Physical Realizable Setting and Figure 1] The abstract states that the framework "is validated by successfully transferring simulated attacks to a physical LiDAR system," and the conclusion states that "simulation can serve as an effective proxy for physical feasibility." However, the "Physical Realizable Setting" section explicitly says that "we do not include real-world testing as part of our formal evaluation protocol," and Figure 1 provides only qualitative camera images and LiDAR point clouds with no detection outputs, bounding boxes, confidence scores, or ASR values. There is therefore no quantitative evidence linking simulation performance to physical attack success. This is load-bearing for the paper's main claim. The authors should either add a quantitative physical experiment with a fixed detector, protocol, and reported ASR, or revise the abstract and conclusion to describe the physical demonstration as illustrative and not as validation.
- [Experimental Settings, Dataset] For the CarLA evaluation, the paper states: "Since it only provides semantic labels, we first use the victim model to generate 3D bounding boxes from clean data. Ground truth boxes are those that overlap with 'Car' points by more than 70%." This makes the CarLA ASR self-referential: the victim model's own clean predictions define the ground truth, so if those predictions are biased, the reported ASR can be inflated without true detection failure. This affects the dataset-transfer results in Table 2 and the simulation-scenario results in Table 5. The authors should obtain independent labels, for example by deriving boxes from the provided semantic point clouds with a non-victim heuristic or an independent detector, or should report agreement between the victim-model-generated labels and another source before using them as ground truth.
- [Experimental Comparison, White-Box Setting, Table 2] The narrative claims that the proposed attack "significantly outperforms PhyAdv," but Table 2 does not uniformly support this. On KITTI PVR, PhyAdv achieves a 3D ASR of 6.99 while the plain "Vanilla" sphere already achieves 21.50 and the proposed MR(8)/MR(9) achieve 31.98/30.69; on CarLA PVR and SC, Vanilla achieves the highest BEV/3D ASR (37.92/41.21 and 43.87/46.98, respectively). The text should acknowledge that a non-adversarial sphere can already induce substantial misdetection in some models and datasets, and should discuss when the optimized object actually improves over the vanilla initialization. This is important for fair-comparison claims because the baseline effect is large in several cells.
minor comments (8)
- [Misrecognization Loss] The heading "Misrecognization Loss" appears to be a typo for "Misrecognition Loss," which is also the term used elsewhere in the paper; please standardize.
- [Equation (5)] The summation in Eq. (5) is written as a sum over k without defining k or specifying whether it ranges over proposals, ground-truth boxes, or matched pairs; please clarify the notation.
- [Equation (11)] For the BEV area metric, the notation with a union over projected triangles is ambiguous: if projected triangles overlap, the union area is not the sum of triangle areas. Please specify whether the computation is the area of the union or the sum of individual projected areas.
- [Table 3] The header row of Table 3 contains a formatting artifact, "3D BoxBEV3D Box," which should be corrected to properly separate the BEV and 3D columns for the target models.
- [Figure 3] The legend entries "PP-NA," "PR-A," and "VR-A" are not defined in the caption; please explain that "NA" and "A" refer to no-attack and attack conditions, respectively.
- [Rendering] The phrase "keep only the top 10 lines" should clarify that "lines" refers to LiDAR scan-line elevations, and the resulting point density should be described, as this preprocessing choice could affect both benign and adversarial detection performance.
- [Challenges] The paper states that reflectivity is neutralized by setting it to zero for all points; this is a significant simplification for physical realizability and should be stated prominently in the evaluation protocol, not only in the Challenges section.
- [Physical Realizable Setting] The physical experiment described in Figure 1 lacks technical details such as the LiDAR sensor model, the distance and orientation of the adversarial object, the number of runs, and the detector used. If the physical demonstration is retained, these details are necessary for reproducibility; if the claim is only qualitative, this should be stated explicitly.
Circularity Check
CarLA pseudo-ground-truth from the victim detector makes part of the simulation benchmark self-referential; KITTI results remain independent.
-
self definitional
[Experimental Settings, Dataset; CarLA rows in Tables 2 and 5]
"Since it only provides semantic labels, we first use the victim model to generate 3D bounding boxes from clean data. Ground truth boxes are those that overlap with “Car” points by more than 70%."
Attack success rate is defined as ASR := (po−pa)/po, where po and pa are detector precisions measured against ground-truth boxes. Here the CarLA ground truth is generated by the victim model itself, which in the white-box protocol is the same target detector g being attacked (Algorithm 1, lines 3-7). The clean precision po therefore scores the detector against its own clean outputs, and pa scores how far attacked outputs deviate from those self-generated labels. The resulting CarLA-based transferability and simulation-scenario numbers quantify self-consistency of the target model rather than genuine detection failure, so that branch of the evaluation is partially fixed by the model's own predictions.
full rationale
Only one genuinely circular step was found, and it is confined to the CarLA branch of the evaluation. The KITTI-based measurements use external ground-truth labels from the KITTI benchmark and are not circular; the white-box, transferability, and black-box tables (Tables 2-4) rest on those external labels. Because CarLA provides only semantic labels, the paper synthesizes 3D bounding-box ground truth with the victim model itself, and ASR is then computed relative to those self-generated boxes, making clean-state mAP a measure of self-agreement rather than true detection quality. This is a partial, not total, circularity: it affects the cross-dataset transfer and CarLA simulation-scenario evidence (Table 5), but the core KITTI attack results remain independent. The abstract's physical-validation claim ('validated by successfully transferring simulated attacks to a physical LiDAR system') is not supported quantitatively and is explicitly qualified away in 'Physical Realizable Setting' by 'we do not include real-world testing as part of our formal evaluation protocol'; that is an evidential gap and internal tension, not a circular reduction, so it does not raise the circularity score further. No load-bearing self-citation was found: the overlapping-author citation (Zhang et al. 2024) appears only as an example of digital point-cloud attacks. Score 5 reflects the one partially self-definitional evaluation component while the central KITTI benchmark retains independent content.
Assumptions & free parameters
free parameters (6)
- Step size epsilon =
PP: 0.005, PR: 0.05, PVR/VR/SC: 0.0005
- Sphere level nu =
2 (162 vertices)
- Initial sphere scale b =
(0.7, 0.7, 0.5) meters
- Laplacian weight lambda =
0.001
- Maximum vertex offset c =
(0.1, 0.1, 0.0) meters
- CarLA ground-truth IoU threshold =
0.7
assumptions (5)
- standard math Möller-Trumbore ray-triangle intersection accurately simulates LiDAR point generation from a mesh.
- domain assumption Simulating only the top 10 LiDAR lines (elevation +2° to -2.19°) preserves the relevant attack surface for rooftop-placed objects.
- domain assumption A sphere initialized on the car rooftop is a representative and sufficient initialization for physical adversarial objects.
- domain assumption Physical feasibility is adequately enforced by bounding-box constraints and Laplacian smoothness.
- ad hoc to paper The victim model's clean predictions on CarLA can serve as valid ground truth for evaluating attacks.
Cite this review
Pith. "Pith review of Revisiting Physically Realizable Adversarial Object Attack against LiDAR-based Detection: Clarifying Problem Formulation and Experimental Protocols." pith.science (2026). https://pith.science/paper/JQXH7BJQ
@misc{pith2026250718457,
author = {Pith},
title = {Pith review of: Revisiting Physically Realizable Adversarial Object Attack against LiDAR-based Detection: Clarifying Problem Formulation and Experimental Protocols},
year = {2026},
howpublished = {\url{https://pith.science/paper/JQXH7BJQ}},
note = {Machine review of arXiv:2507.18457}
}
read the original abstract
Adversarial robustness in LiDAR-based 3D object detection is a critical research area due to its widespread application in real-world scenarios. While many digital attacks manipulate point clouds or meshes, they often lack physical realizability, limiting their practical impact. Physical adversarial object attacks remain underexplored and suffer from poor reproducibility due to inconsistent setups and hardware differences. To address this, we propose a device-agnostic, standardized framework that abstracts key elements of physical adversarial object attacks, supports diverse methods, and provides open-source code with benchmarking protocols in simulation and real-world settings. Our framework enables fair comparison, accelerates research, and is validated by successfully transferring simulated attacks to a physical LiDAR system. Beyond the framework, we offer insights into factors influencing attack success and advance understanding of adversarial robustness in real-world LiDAR perception.
Figures
Reference graph
Works this paper leans on
-
[1]
, " * write output.state after.block = add.period write newline
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...
-
[2]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
-
[3]
Abdelfattah, M.; Yuan, K.; Wang, Z. J.; and Ward, R. 2021. Towards universal physical attacks on cascaded camera-lidar 3d object detection models. In 2021 IEEE International Conference on Image Processing (ICIP), 3592--3596. IEEE
work page 2021
-
[4]
Aung, N. H. H.; Sangwongngam, P.; Jintamethasawat, R.; Shah, S.; and Wuttisittikulkij, L. 2024. A review of lidar-based 3d object detection via deep learning approaches towards robust connected and autonomous vehicles. IEEE Transactions on Intelligent Vehicles
work page 2024
-
[5]
Cao, Y.; Xiao, C.; Cyr, B.; Zhou, Y.; Park, W.; Rampazzi, S.; Chen, Q. A.; Fu, K.; and Mao, Z. M. 2019 a . Adversarial sensor attack on lidar-based perception in autonomous driving. In Proceedings of the 2019 ACM SIGSAC conference on computer and communications security, 2267--2281
work page 2019
-
[6]
Cao, Y.; Xiao, C.; Yang, D.; Fang, J.; Yang, R.; Liu, M.; and Li, B. 2019 b . Adversarial objects against lidar-based autonomous driving systems. arXiv preprint arXiv:1907.05418
arXiv 2019
-
[7]
Deng, J.; Shi, S.; Li, P.; Zhou, W.; Zhang, Y.; and Li, H. 2021. Voxel r-cnn: Towards high performance voxel-based 3d object detection. In Proceedings of the AAAI conference on artificial intelligence, volume 35, 1201--1209
2021
-
[8]
Deschaud , J.-E. 2021. KITTI-CARLA: a KITTI-like dataset generated by CARLA Simulator . arXiv e-prints, arXiv:2109.00892
work page Pith review arXiv 2021
Show all 35 references
-
[9]
Engelmann, F.; St \"u ckler, J.; and Leibe, B. 2017. SAMP: shape and motion priors for 4d vehicle reconstruction. In 2017 IEEE Winter Conference on Applications of Computer Vision (WACV), 400--408. IEEE
2017
-
[10]
Geiger, A.; Lenz, P.; and Urtasun, R. 2012. Are we ready for autonomous driving? the kitti vision benchmark suite. In 2012 IEEE conference on computer vision and pattern recognition, 3354--3361. IEEE
2012
-
[11]
Huang, Q.; Dong, X.; Chen, D.; Zhou, H.; Zhang, W.; and Yu, N. 2022. Shape-invariant 3D Adversarial Point Clouds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 15335--15344
2022
-
[12]
Ju, Z.; Zhang, H.; Li, X.; Chen, X.; Han, J.; and Yang, M. 2022. A survey on attack detection and resilience for connected and automated vehicles: From vehicle dynamics and control perspective. IEEE Transactions on Intelligent Vehicles, 7(4): 815--837
2022
-
[13]
J.; and Bengio, S
Kurakin, A.; Goodfellow, I. J.; and Bengio, S. 2018. Adversarial examples in the physical world. In Artificial intelligence safety and security, 99--112. Chapman and Hall/CRC
2018
-
[14]
H.; Vora, S.; Caesar, H.; Zhou, L.; Yang, J.; and Beijbom, O
Lang, A. H.; Vora, S.; Caesar, H.; Zhou, L.; Yang, J.; and Beijbom, O. 2019. Pointpillars: Fast encoders for object detection from point clouds. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 12697--12705
2019
-
[15]
Li, J.; Schmidt, F.; and Kolter, Z. 2019. Adversarial camera stickers: A physical camera-based attack on deep learning systems. In International conference on machine learning, 3896--3904. PMLR
2019
-
[16]
M \"o ller, T.; and Trumbore, B. 2005. Fast, minimum storage ray/triangle intersection. In ACM SIGGRAPH 2005 Courses, 7--es
2005
-
[17]
Ren, H.; Huang, T.; and Yan, H. 2021. Adversarial examples: attacks and defenses in the physical world. International Journal of Machine Learning and Cybernetics, 12(11): 3325--3336
2021
-
[18]
Shi, S.; Guo, C.; Jiang, L.; Wang, Z.; Shi, J.; Wang, X.; and Li, H. 2020. Pv-rcnn: Point-voxel feature set abstraction for 3d object detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 10529--10538
2020
-
[19]
Shi, S.; Wang, X.; and Li, H. 2019. Pointrcnn: 3d object proposal generation and detection from point cloud. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 770--779
2019
-
[20]
Song, X.; Wang, P.; Zhou, D.; Zhu, R.; Guan, C.; Dai, Y.; Su, H.; Li, H.; and Yang, R. 2019. Apollocar3d: A large 3d car instance understanding benchmark for autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 5452--5462
2019
-
[21]
Song, Z.; Liu, L.; Jia, F.; Luo, Y.; Jia, C.; Zhang, G.; Yang, L.; and Wang, L. 2024. Robustness-aware 3d object detection in autonomous driving: A review and outlook. IEEE Transactions on Intelligent Transportation Systems
2024
-
[22]
A.; and Mao, Z
Sun, J.; Cao, Y.; Chen, Q. A.; and Mao, Z. M. 2020. Towards robust \ LiDAR-based \ perception in autonomous driving: General black-box adversarial sensor attack and countermeasures. In 29th USENIX Security Symposium (USENIX Security 20), 877--894
2020
-
[23]
Team, O. D. 2020. OpenPCDet: An Open-source Toolbox for 3D Object Detection from Point Clouds. https://github.com/open-mmlab/OpenPCDet
2020
-
[24]
Thys, S.; Van Ranst, W.; and Goedem \'e , T. 2019. Fooling automated surveillance cameras: adversarial patches to attack person detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, 0--0
2019
-
[25]
Tu, J.; Ren, M.; Manivasagam, S.; Liang, M.; Yang, B.; Du, R.; Cheng, F.; and Urtasun, R. 2020. Physically realizable adversarial examples for lidar object detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 13716--13725
2020
-
[26]
Wang, J.; Li, F.; Zhang, X.; and Sun, H. 2023. Adversarial Obstacle Generation against LiDAR-Based 3D Object Detection. IEEE Transactions on Multimedia
2023
-
[27]
Wen, Y.; Lin, J.; Chen, K.; Chen, C.; and Jia, K. 2019. Geometry-aware generation of adversarial point clouds. arXiv preprint arXiv:1912.11171
2019 arXiv
-
[28]
Yan, Y.; Mao, Y.; and Li, B. 2018. Second: Sparsely embedded convolutional detection. Sensors, 18(10): 3337
2018
-
[29]
Zhang, H.; Cheng, L.; He, Q.; Huang, W.; Li, R.; Sicre, R.; Huang, X.; Hermanns, H.; and Zhang, L. 2024. Eidos: Efficient, Imperceptible Adversarial 3D Point Clouds. arXiv preprint arXiv:2405.14210
2024 arXiv
-
[30]
Zhang, H.; Furon, T.; Amsaleg, L.; and Avrithis, Y. 2022. Deep Neural Network Attacks and Defense: The Case of Image Classification. Multimedia Security, 1: 41--75
2022
-
[31]
Zhang, J.; Chen, L.; Liu, B.; Ouyang, B.; Xie, Q.; Zhu, J.; Li, W.; and Meng, Y. 2021. 3d adversarial attacks beyond point cloud. arXiv preprint arXiv:2104.12146
2021 arXiv
-
[32]
Zhang, Y.; Hou, J.; and Yuan, Y. 2024. A comprehensive study of the robustness for lidar-based 3d object detectors against adversarial attacks. International Journal of Computer Vision, 132(5): 1592--1624
2024
-
[33]
Zhao, Z.; Liu, Z.; and Larson, M. 2021. On success and simplicity: A second look at transferable targeted attacks. Advances in Neural Information Processing Systems, 34: 6115--6128
2021
-
[34]
Zhou, Z.; Tang, D.; Wang, X.; Han, W.; Liu, X.; and Zhang, K. 2018. Invisible mask: Practical attacks on face recognition with infrared. arXiv preprint arXiv:1803.04683
2018 arXiv
-
[35]
Zhu, Y.; Miao, C.; Zheng, T.; Hajiaghajani, F.; Su, L.; and Qiao, C. 2021. Can we use arbitrary objects to attack lidar perception in autonomous driving? In Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security, 1945--1960
2021
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.