REVIEW 3 major objections 4 minor 36 references
Assured Autonomy with Neuro-Symbolic Perception
T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Scene graphs expose a LiDAR attack that fools camera–LiDAR fusion.
desk verdict A plausible integration of scene-graph consistency for attack detection, but the feasibility evidence is anecdotal and the abstract overclaims; the idea deserves a serious referee with major revision. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is cross-sensor scene graph consistency evaluation. A scene graph is a graph whose nodes are detected objects and whose edges are relational predicates such as front of, left of, near, and occluding; scene graph generation (SGG) is the joint task of detecting objects and inferring these relations. For the camera, graphs are produced by a vision-language foundation model prompted in natural language; for LiDAR, graphs are produced by hand-coded geometric functions over 3D box detections. Per-sensor integrity checks each graph against physics- and commonsense-based knowledge, and a cross-sensor integrity stage matches nodes between graphs and flags edges that disagree. This graph-level comparison is what lifts 2D image data into relational 3D space and exposes semantic changes that box-level IoU checks cannot see.
What would settle it
Run the pipeline on a large set of benign CARLA and nuScenes scenes and count cross-sensor inconsistencies; if the camera foundation-model graph and the LiDAR rule-based graph disagree frequently on un-attacked scenes, high false-alarm rates would mean the proposed comparison cannot distinguish attack-induced semantic changes from ordinary model noise.
Extended reading notes
Core claim
On the paper's own terms, the central discovery is that representing sensor output as relational scene graphs turns a previously stealthy attack into a detectable semantic inconsistency. A frustum attack translates an object's 3D box (for example, a van) so the projected box keeps high IoU with the unattacked camera detection; standard single-frame fusion checks therefore pass. But the attack moves the van in 3D space, changing its geometric relations to other objects. The LiDAR rule-based graph then shows, for example, that the van-truck relation flips, while the camera graph, built by a foundation model prompted to "build a scene graph from this image," preserves the original spatial relation. Comparing the two graphs exposes the mismatch, so the authors claim the neuro-symbolic SGG pipeline offers the first method securing perception against attacks that exploit asymmetric sensor resolutions.
Load-bearing premise
The camera graph must be a trustworthy reference: the foundation model's scene graph has to be accurate, stable, and comparable to the LiDAR geometric graph, but the paper does not measure how often the two graphs agree on benign scenes.
Editorial extensions
If this is right
- Frustum-type translation attacks on LiDAR become detectable even when the attacked boxes keep high overlap with the unattacked camera image.
- The same graph-based integrity reasoning can flag semantic-structure attacks beyond translations, such as false positives or false negatives that alter relational context.
- Graph-informed fusion can use per-node and per-edge consistency classifications to weight or reject unreliable sensor updates during state estimation.
- If specialized transformer-based SGG models replace foundation models, the approach can move toward real-time deployment with bounded output spaces.
Reading between the lines
- Editorial inference: the same cross-sensor graph comparison could extend to other modalities such as radar or thermal cameras, since any sensor that can produce relational structure could be checked against the camera-derived graph.
- Editorial inference: the approach could also detect spoofed objects that are internally consistent with one sensor but break relational constraints across sensors, not only translated existing objects.
- Editorial inference: a quantitative benign-scene false-alarm study is the natural next experiment, because the practical detection threshold of the graph comparison is undefined without measuring how often the two graph types naturally disagree.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes NeuSPaPer, a neuro-symbolic perception framework that augments object detection with scene graph generation (SGG) and uses per-sensor and cross-sensor integrity checks to detect semantic-structure attacks, in particular the frustum attack on LiDAR-camera fusion. The authors derive an optimal frustum attack (Eq. 1), propose rule-based SGG for LiDAR and foundation-model SGG for camera images, and present a qualitative case study from CARLA and two from nuScenes. They claim that this is the first single-platform method to detect such attacks, and they discuss challenges and future work toward full-stack neuro-symbolic perception.
Significance. The idea of using scene graphs as a relational integrity layer for multi-sensor perception is attractive, and the attack formulation in Eq. (1) is consistent with prior work. If the central claim were established with quantitative evidence, this would be a meaningful step toward defending fusion pipelines against semantic-structure attacks. However, the evaluated evidence is only qualitative, and the unmeasured camera branch makes the core comparison untested. The paper is best viewed as a research vision with a feasibility illustration, not as a validated security mechanism.
major comments (3)
- [Section 5.3, Fig. 6] The central claim that the neuro-symbolic SGG pipeline 'offers the first method that secures perception against attacks exploiting asymmetric sensor resolutions' is not supported by the reported feasibility study, which contains no detection rate, no false-positive analysis, no benign-scene baseline, and no error bars. The demonstration is a single CARLA frame plus two nuScenes frames in Appendix D, and the inconsistency highlighted in Fig. 6(d) is not compared against the distribution of graph mismatches that would occur in unattacked scenes. Without such a comparison, the observed mismatch cannot be attributed to the frustum attack rather than to ordinary sensor noise, detection error, or vocabulary differences between the two graph types.
- [Section 5.1, Figure 4 caption, Appendix C.3] The cross-sensor consistency check requires that the foundation-model scene graph from the camera and the rule-based scene graph from LiDAR are commensurable, but the paper never provides an ontology mapping between the foundation model's relation vocabulary and the geometric rules (front_of, left_of, occluding, following, far_from, close_to, next_to). The Figure 4 caption states that 'node and edge types differ from rule-based approach,' and no benign-scene study demonstrates that the two graphs normally agree. If the foundation model says 'beside' where the rule-based graph says 'front_of,' or fails to detect the distant truck in Fig. 6(a), the cross-sensor evaluator will flag benign data as attack, so the load-bearing assertion of Section 5.3 is unverified.
- [Section 7] The conclusion concedes that 'uncertain or incomplete graphs due to noisy data can yield inaccurate conclusions,' which directly undermines the feasibility demonstration, because the paper does not quantify graph noise on benign scenes or show that the cross-sensor integrity function separates noise from attack. This concession should either be backed by a false-alarm analysis on benign frames or the Section 5.3 claim should be downgraded from 'first method that secures perception' to a preliminary illustration.
minor comments (4)
- [Appendix D] The text says 'Additional analyses and case studies are provided in Appendix D,' but Appendix D contains only two additional qualitative case studies with no numeric analysis; please align the wording with the content.
- [Section 5.1] The foundation model is never named; to make the study reproducible, specify the model, version, prompt details, and any sampling parameters.
- [Appendix C.3] The list of relations includes 'far from' and 'close to' with complement 'self'; the notion of a 'reduced graph' and how complements are handled for these self-complement relations should be clarified.
- [Section 3.1] The statement that 'no amount of sub-sampling or ensembling can mitigate translation attacks' is too categorical; the paper should precisely state the class of defenses it rules out and cite the relevant impossibility or possibility results.
Circularity Check
No significant circularity: the cross-sensor graph comparison uses an independently pre-trained foundation model plus hand-coded geometric rules, so the attack-induced mismatch is the intended detection mechanism rather than an input-output tautology; the remaining concerns are validation gaps, not circular reasoning.
full rationale
The paper's central demonstration (Secs. 5.1-5.3, Figs. 6-8) does not fit any parameter to the attack and does not present a fitted quantity as a prediction. The LiDAR branch is a deterministic rule-based scene graph over 3D boxes (Sec. 5.1 and Appendix C.3); the camera branch is an externally pre-trained vision-language foundation model prompted with 'Build a scene graph from this image' (Sec. 5.1), which the authors neither train nor tune. The frustum attack is re-derived in Eq. (1) of Appendix B.4 rather than being assumed from a self-citation. The attack's 3D box translation necessarily alters the hand-coded geometric relations; flagging that change through cross-sensor edge matching (Sec. 4.2.3) is the intended operation of the proposed detector, not a circular prediction. The paper does not claim the graph mismatch is a learned or fitted output; it is a feasibility observation. The genuine weaknesses -- no benign-scene false-alarm baseline, unmeasured SGG accuracy of the foundation model, and the acknowledged 'node and edge types differ' between graph modalities (Fig. 4 caption; Sec. 7: 'uncertain or incomplete graphs due to noisy data can yield inaccurate conclusions') -- are empirical validation gaps, not circular reasoning. Self-citations to Hallyburton et al. (2022, 2023a) supply the attack and stealth framework, but the formal optimization is restated in the paper, so they are not load-bearing circular evidence.
Assumptions & free parameters
free parameters (3)
- Spatial-relation thresholds for rule-based LiDAR graphs =
not reported (deferred to unreleased code)
- Cross-sensor graph consistency tolerance =
not reported
- IoU stealth threshold zeta_min for the frustum attack =
about 0.9 in Figure 1 (from prior work)
assumptions (5)
- domain assumption Point-cloud perception networks are translation equivariant, so features of a translated object are retained at the new location.
- domain assumption Certified robustness guarantees are limited to bounded Lp perturbations and therefore do not cover semantic relocation attacks.
- domain assumption A vision-language foundation model prompted with 'Build a scene graph from this image' yields spatial relationships that are reliable and commensurable with geometric relations computed from 3D boxes.
- standard math The assignment-problem and IoU-projection formulations (Appendices A and B) correctly model camera-LiDAR fusion and the frustum attack.
- ad hoc to paper In the case studies, benign camera and LiDAR graphs agree, so a graph mismatch is an attack signal rather than sensor noise.
Cite this review
Pith. "Pith review of Assured Autonomy with Neuro-Symbolic Perception." pith.science (2026). https://pith.science/paper/FLTLQV3X
@misc{pith2026250521322,
author = {Pith},
title = {Pith review of: Assured Autonomy with Neuro-Symbolic Perception},
year = {2026},
howpublished = {\url{https://pith.science/paper/FLTLQV3X}},
note = {Machine review of arXiv:2505.21322}
}
read the original abstract
Many state-of-the-art AI models deployed in cyber-physical systems (CPS), while highly accurate, are simply pattern-matchers.~With limited security guarantees, there are concerns for their reliability in safety-critical and contested domains. To advance assured AI, we advocate for a paradigm shift that imbues data-driven perception models with symbolic structure, inspired by a human's ability to reason over low-level features and high-level context. We propose a neuro-symbolic paradigm for perception (NeuSPaPer) and illustrate how joint object detection and scene graph generation (SGG) yields deep scene understanding.~Powered by foundation models for offline knowledge extraction and specialized SGG algorithms for real-time deployment, we design a framework leveraging structured relational graphs that ensures the integrity of situational awareness in autonomy. Using physics-based simulators and real-world datasets, we demonstrate how SGG bridges the gap between low-level sensor perception and high-level reasoning, establishing a foundation for resilient, context-aware AI and advancing trusted autonomy in CPS.
Figures
Figures from the paper (5 more)
Reference graph
Works this paper leans on
-
[1]
Architecting artificial intelligence for autonomous cars: The openpilot framework
Luciano Baresi and Damian A Tamburri. Architecting artificial intelligence for autonomous cars: The openpilot framework. In European Conference on Software Architecture, pages 189--204. Springer, 2023
work page 2023
-
[2]
Geometric deep learning: Grids, groups, graphs, geodesics, and gauges
Michael M Bronstein, Joan Bruna, Taco Cohen, and Petar Veli c kovi \'c . Geometric deep learning: Grids, groups, graphs, geodesics, and gauges. arXiv preprint arXiv:2104.13478, 2021
arXiv 2021
-
[3]
nuscenes: A multimodal dataset for autonomous driving
Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11621--11631, 2020
2020
-
[4]
Adversarial sensor attack on lidar-based perception in autonomous driving
Yulong Cao, Chaowei Xiao, Benjamin Cyr, Yimeng Zhou, Won Park, Sara Rampazzi, Qi Alfred Chen, Kevin Fu, and Z Morley Mao. Adversarial sensor attack on lidar-based perception in autonomous driving. In Proceedings of the 2019 ACM SIGSAC conference on computer and communications security, pages 2267--2281, 2019
work page 2019
-
[5]
End-to-end object detection with transformers
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision, pages 213--229. Springer, 2020
2020
-
[6]
Adversarial examples are not easily detected: Bypassing ten detection methods
Nicholas Carlini and David Wagner. Adversarial examples are not easily detected: Bypassing ten detection methods. In Proceedings of the 10th ACM workshop on artificial intelligence and security, pages 3--14, 2017
work page 2017
-
[7]
A review: Knowledge reasoning over knowledge graph
Xiaojun Chen, Shengbin Jia, and Yang Xiang. A review: Knowledge reasoning over knowledge graph. Expert systems with applications, 141: 0 112948, 2020
work page 2020
-
[8]
Scene Graph Generation in Autonomous Driving: a Neuro-symbolic approach
Paolo Emmanuel Ilario Dimasi. Scene Graph Generation in Autonomous Driving: a Neuro-symbolic approach. PhD thesis, Politecnico di Torino, 2023
work page 2023
Show all 36 references
-
[9]
Carla: An open urban driving simulator
Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. Carla: An open urban driving simulator. In Conference on robot learning, pages 1--16. PMLR, 2017
2017
-
[10]
Multisensor data fusion
Hugh Durrant-Whyte and Thomas C Henderson. Multisensor data fusion. Springer handbook of robotics, pages 867--896, 2016
2016
-
[11]
Robust physical-world attacks on deep learning visual classification
Kevin Eykholt, Ivan Evtimov, Earlence Fernandes, Bo Li, Amir Rahmati, Chaowei Xiao, Atul Prakash, Tadayoshi Kohno, and Dawn Song. Robust physical-world attacks on deep learning visual classification. In Proceedings of the IEEE conference on computer vision and pattern recognit...
2018
-
[12]
Uav-based multi-sensor data fusion and machine learning algorithm for yield prediction in wheat
Shuaipeng Fei, Muhammad Adeel Hassan, Yonggui Xiao, Xin Su, Zhen Chen, Qian Cheng, Fuyi Duan, Riqiang Chen, and Yuntao Ma. Uav-based multi-sensor data fusion and machine learning algorithm for yield prediction in wheat. Precision agriculture, 24 0 (1): 0 187--212, 2023
2023
-
[13]
Adversarial attacks on medical machine learning
Samuel G Finlayson, John D Bowers, Joichi Ito, Jonathan L Zittrain, Andrew L Beam, and Isaac S Kohane. Adversarial attacks on medical machine learning. Science, 363 0 (6433): 0 1287--1289, 2019
2019
-
[14]
Neuro-symbolic reasoning in the traffic domain
Leilani H Gilpin and Filip Ilievski. Neuro-symbolic reasoning in the traffic domain. J AI Res, 15 0 (3): 0 123--145, 2021
2021
-
[15]
Road-r: the autonomous driving dataset with logical requirements
Eleonora Giunchiglia, Mihaela C a t a lina Stoian, Salman Khan, Fabio Cuzzolin, and Thomas Lukasiewicz. Road-r: the autonomous driving dataset with logical requirements. Machine Learning, 112 0 (9): 0 3261--3291, 2023
2023
-
[16]
Datasets, models, and algorithms for multi-sensor, multi-agent autonomy using avstack
R Spencer Hallyburton and Miroslav Pajic. Datasets, models, and algorithms for multi-sensor, multi-agent autonomy using avstack. arXiv preprint arXiv:2312.04970, 2023
2023 arXiv
-
[17]
Security analysis of camera-lidar fusion against black-box attacks on autonomous vehicles
R Spencer Hallyburton, Yupei Liu, Yulong Cao, Z Morley Mao, and Miroslav Pajic. Security analysis of camera-lidar fusion against black-box attacks on autonomous vehicles. In 31st USENIX Security Symposium (USENIX Security 22), pages 1903--1920, 2022
1903
-
[18]
Partial-information, longitudinal cyber attacks on lidar in autonomous vehicles
R Spencer Hallyburton, Qingzhao Zhang, Z Morley Mao, and Miroslav Pajic. Partial-information, longitudinal cyber attacks on lidar in autonomous vehicles. arXiv preprint arXiv:2303.03470, 2023 a
2023 arXiv
-
[19]
Avstack: An open-source, reconfigurable platform for autonomous vehicle development
Robert Spencer Hallyburton, Shucheng Zhang, and Miroslav Pajic. Avstack: An open-source, reconfigurable platform for autonomous vehicle development. In Proceedings of the ACM/IEEE 14th International Conference on Cyber-Physical Systems (with CPS-IoT Week 2023), pages 209--220, 2023 b
2023
-
[20]
Egtr: Extracting graph from transformer for scene graph generation
Jinbae Im, JeongYeon Nam, Nokyung Park, Hyungmin Lee, and Seunghyun Park. Egtr: Extracting graph from transformer for scene graph generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24229--24238, 2024
2024
-
[21]
Certified robustness for top-k predictions against adversarial perturbations via randomized smoothing
Jinyuan Jia, Xiaoyu Cao, Binghui Wang, and Neil Zhenqiang Gong. Certified robustness for top-k predictions against adversarial perturbations via randomized smoothing. arXiv preprint arXiv:1912.09899, 2019
1912 arXiv
-
[22]
Image retrieval using scene graphs
Justin Johnson, Ranjay Krishna, Michael Stark, Li-Jia Li, David Shamma, Michael Bernstein, and Li Fei-Fei. Image retrieval using scene graphs. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3668--3678, 2015
2015
-
[23]
Vilt: Vision-and-language transformer without convolution or region supervision
Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Vision-and-language transformer without convolution or region supervision. In International conference on machine learning, pages 5583--5594. PMLR, 2021
2021
-
[24]
Pointpillars: Fast encoders for object detection from point clouds
Alex H Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom. Pointpillars: Fast encoders for object detection from point clouds. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12697--12705, 2019
2019
-
[25]
Sgtr: End-to-end scene graph generation with transformer
Rongjie Li, Songyang Zhang, and Xuming He. Sgtr: End-to-end scene graph generation with transformer. In proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 19486--19496, 2022
2022
-
[26]
Visual relationship detection with language priors
Cewu Lu, Ranjay Krishna, Michael Bernstein, and Li Fei-Fei. Visual relationship detection with language priors. In Computer Vision--ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11--14, 2016, Proceedings, Part I 14, pages 852--869. Springer, 2016
2016
-
[27]
Distillation as a defense to adversarial perturbations against deep neural networks
Nicolas Papernot, Patrick McDaniel, Xi Wu, Somesh Jha, and Ananthram Swami. Distillation as a defense to adversarial perturbations against deep neural networks. In 2016 IEEE symposium on security and privacy (SP), pages 582--597. IEEE, 2016
2016
-
[28]
Potential cyberattacks on automated vehicles
Jonathan Petit and Steven E Shladover. Potential cyberattacks on automated vehicles. IEEE Transactions on Intelligent transportation systems, 16 0 (2): 0 546--556, 2014
2014
-
[29]
Learning transferable visual models from natural language supervision
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pa...
2021
-
[30]
You only look once: Unified, real-time object detection
J Redmon. You only look once: Unified, real-time object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2016
2016
-
[31]
Faster r-cnn: Towards real-time object detection with region proposal networks
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. IEEE transactions on pattern analysis and machine intelligence, 39 0 (6): 0 1137--1149, 2016
2016
-
[32]
Universal adversarial training
Ali Shafahi, Mahyar Najibi, Zheng Xu, John Dickerson, Larry S Davis, and Tom Goldstein. Universal adversarial training. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 5636--5643, 2020
2020
-
[33]
Pv-rcnn: Point-voxel feature set abstraction for 3d object detection
Shaoshuai Shi, Chaoxu Guo, Li Jiang, Zhe Wang, Jianping Shi, Xiaogang Wang, and Hongsheng Li. Pv-rcnn: Point-voxel feature set abstraction for 3d object detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10529--10538, 2020
2020
-
[34]
Unbiased scene graph generation from biased training
Kaihua Tang, Yulei Niu, Jianqiang Huang, Jiaxin Shi, and Hanwang Zhang. Unbiased scene graph generation from biased training. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3716--3725, 2020
2020
-
[35]
Physically realizable adversarial examples for lidar object detection
James Tu, Mengye Ren, Sivabalan Manivasagam, Ming Liang, Bin Yang, Richard Du, Frank Cheng, and Raquel Urtasun. Physically realizable adversarial examples for lidar object detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 13...
2020
-
[36]
Neural motifs: Scene graph parsing with global context
Rowan Zellers, Mark Yatskar, Sam Thomson, and Yejin Choi. Neural motifs: Scene graph parsing with global context. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5831--5840, 2018
2018
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.