REVIEW 4 major objections 4 minor 35 references
Leveraging Trustworthy AI for Automotive Security in Multi-Domain Operations: Towards a Responsive Human-AI Multi-Domain Task Force for Cyber Social Security
T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read The paper argues that Random Forest and Gradient Boosting can be configured so that the ZOO black-box attack takes much longer to succeed, while XGB cannot.
desk verdict A straightforward empirical extension of the authors' own prior ZOO-attack study: the RQ1 timing measurements are credible and potentially useful, but the RQ2 window-of-opportunity claim rests on a missing measurement of post-adversarial-training attack time. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the dependence of ZOO attack runtime on model ensemble size: each adversarial example requires many numerical gradient-estimation queries through the victim model, so a model with more trees or boosting rounds takes longer to evaluate. This turns a hyperparameter (number of estimators) into an intrinsic defense, measured by the slope of attack time versus estimator count under linear regression.
What would settle it
Run the same ZOO attack-time experiment with the same three models but on a different split of OTIDS or on another CAN-bus dataset, such as the Survival or Car Hacking datasets; if attack time does not increase with tree or boosting-round counts there, the intrinsic-defense claim fails. A second check is to see whether XGB shows a linear increase under different ZOO learning-rate settings.
Extended reading notes
Core claim
The paper reports that, for a pure black-box ZOO attack against supervised machine-learning-based intrusion detection on CAN bus frames, the time needed to produce 92,270 adversarial examples grows linearly with the number of bagging trees in Random Forest and with the number of boosting rounds in Gradient Boosting, while Extreme Gradient Boosting shows no such consistent relationship. On this basis the paper treats tree-count and boosting-round settings as intrinsic defensive mechanisms, and shows that adversarial training time is comparatively small, so a defender can combine these settings with adversarial training to widen the window in which a Human-AI task force can detect an intrusion.
Load-bearing premise
The timing measurements are taken only on one subset (B') of one CAN-bus dataset and are assumed to represent the full data distribution, so the reported slowdown from tuning hyperparameters may not generalize to other traffic or other datasets.
Editorial extensions
If this is right
- For Random Forest, the reported linear fit implies each additional bagging tree adds roughly 21 minutes of ZOO attack time for the tested adversarial-example set, so increasing tree count is a tunable way to slow an attacker.
- For Gradient Boosting, each extra boosting round adds about 25 seconds of attack time, giving a weaker but still measurable slowdown.
- For Extreme Gradient Boosting, boosting rounds do not produce a consistent linear increase in attack time, so XGB does not offer the same intrinsic-defense benefit.
- Adversarial training time stays on the order of minutes while attack time is on the order of hours, so the defender's countermeasure is much cheaper than the attacker's effort, and the paper recommends Random Forest as the more defensive choice.
Reading between the lines
- We infer that the same hyperparameter-slowing effect may hold for other gradient-estimation black-box attacks, but the paper only tests ZOO, so this is an extension rather than a result.
- A natural testable extension is to check whether an attacker can compensate by raising ZOO's learning rate or reducing its iteration count, which would reveal whether the defense is robust against adaptive attackers.
- Because the measurements rely on a single dataset subset, replicating the experiment on a different CAN-bus dataset would show whether the qualitative ordering RF and GB sensitive, XGB not persists in other traffic distributions.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper studies whether the number of estimators/boosting rounds in Random Forest (RF), Gradient Boosting (GB), and Extreme Gradient Boosting (XGB) affects the time required to generate adversarial examples with the ZOO black-box attack against an ML-based CAN bus intrusion detection system (IDS). Using the OTIDS dataset and a pipeline inherited from the authors' previous work, the paper measures attack-generation time on a subset B' and adversarial-training (AT) time on the combined training set. It reports a positive linear trend for RF and GB, a non-trend for XGB, and concludes that careful hyperparameter selection combined with AT can reduce the attacker's 'window of opportunity.' It then qualitatively maps this conclusion to the Detection-Response-Prevention axes of the CSS-MDO framework to support educational claims about Human-AI Multi-Domain Task Forces.
Significance. If the empirical claims held, the paper would provide a practically useful, low-cost defensive lever: choosing ensemble sizes can increase the computational cost of black-box adversarial examples without altering the deployed model family. The direct timing measurements, the use of a standard attack library (ART), and the concrete automotive threat model are strengths. However, the significance is substantially limited by the fact that the central operational claim (RQ2) relies on comparing AT training time with attack time measured only on the pre-AT model, and the statistical evidence for the RQ1 trends is thin (no error bars, no goodness-of-fit values, no repetitions). The qualitative RQ3 analysis is appropriately framed as qualitative, but it inherits the weaknesses of the underlying quantitative claims.
major comments (4)
- [§4.1, Figures 2–4] The regression lines in Figures 2–4 are presented with coefficients but no goodness-of-fit statistics, confidence intervals, or repeated trials. The claim that RF and GB show a 'clear linear relationship' while XGB does not is not statistically substantiated. RQ1's affirmative/negative answers hinge on this distinction; please report at least R² or adjusted R², and ideally per-configuration error bars or repeated measurements, for every regression.
- [§4.1, RQ2 (Figures 2–6)] RQ2's affirmative answer is unsupported because attack time is measured only on Model_A (the pre-adversarial-training model), while the defender's AT timing is measured on the hardened Model_A+B+B'. The attacker's time against the AT model is never measured; Section 5 explicitly lists 'attack times following the application of the countermeasure' as future work. Without that measurement, comparing pre-AT attack time with AT training time cannot establish a reduced operational window. This is a load-bearing gap for the paper's D-TAI/MDTF impact narrative.
- [§3.3] The empirical timing study is conducted on subset B' under the stated assumption that B' 'is consistent with C''. No evidence is provided for this assumption: no distributional comparison, no cross-validation on other splits, and no report of whether the timing relationship changes on C'. Since all RQ1 conclusions and the extrapolated attack times (e.g., ~31 hours for 81 RF trees) are derived solely from B', the lack of validation weakens the generality of the results. Please quantify representativeness or repeat the timing study on at least one held-out split.
- [§4.1, final paragraph] The statement that AT time is 'significantly less than the time provided to the H-AI MDTF to detect the intrusion in the CAN network' introduces a detection-time quantity that is never defined or measured in the paper. Without a concrete definition and measurement of intrusion-detection latency, this comparison is unsupported and should be removed or explicitly identified as a hypothesis.
minor comments (4)
- [§5] The paper contains several typos and grammatical errors, including 'realted', 'prelaborated', 'demage', 'coutermeasure', 'consedered', and 'examinated'; a careful proofreading pass is needed.
- [§1 and §2.1] The acronym 'TT' is introduced as 'Technology Transfer' but the models RF/GB/XGB are ordinary supervised classifiers; the connection to technology transfer is not explained and may confuse readers.
- [§4.1, Figure captions] The figure captions and axis labels use 'Estimated Time' for attack and training time, but the estimation method is not described; clarifying whether these are single-run measurements, averages, or fitted values would improve transparency.
- [§4.2, Table 1] The impact ratings ('Very High', 'High') and motivations are self-assessed with no defined rating scale or assessment procedure; the table should be labeled as an illustrative qualitative assessment rather than an empirical evaluation result.
Circularity Check
No circularity: the central timing measurements are direct empirical observations, and the reused prior-work elements are not load-bearing for the new quantitative claims.
full rationale
The paper's core quantitative contribution, RQ1, is a direct measurement of adversarial-example generation time as a function of hyperparameter values (Figures 2-4). These times are measured on Model_A with the ZOO attack and are not derived from, nor defined in terms of, the conclusions they support. RQ2 compares these measured attack times with separately measured adversarial-training times (Figures 5-6); this comparison is methodologically incomplete because post-AT attack times are explicitly left to future work, but that is an evidential gap, not a circular reduction. The claim that the paper reuses the authors' prior pipeline and framework is accurate but not circular: Section 3.2 imports the attack pipeline from Barletta et al. [29] as methodology, and the current paper's new timing data are independently generated. The qualitative CSS-MDO impact assessment in Section 4.2 is a subjective mapping, not a derivation. The stated assumption in Section 3.3 that B' is consistent with C' is a generalizability assumption, not a definitional identity. No equation or fitted parameter is renamed as a prediction, and no load-bearing conclusion is forced by self-citation. Therefore, while there are correctness risks in the RQ2 inference, there is no circularity in the paper's derivation chain.
Assumptions & free parameters
free parameters (3)
- ZOO attack hyperparameters =
learning_rate=0.1, max_iter=50, variable_h=0.2
- Attack-time linear regression coefficients =
RF: slope 1243.65 s/tree, intercept 8876.2 s; GB: slope 24.93 s/round, intercept 6704.1 s; XGB: slope 33.25 s/round…
- Adversarial-training-time regression coefficients =
RF: slope 1.41 s/tree, intercept -14.28 s; GB: slope 6.01 s/round, intercept 57.23 s
assumptions (5)
- domain assumption Subset B' is representative of C' for adversarial example generation time.
- domain assumption Adversarial example generation time is a valid proxy for the attacker's operational cost and window of opportunity.
- domain assumption A single workstation's timing measurements are representative across hardware.
- ad hoc to paper Linear regression without goodness-of-fit statistics is sufficient to infer sensitivity.
- domain assumption Pure black-box attacker can obtain true labels through a compromised ECU or IDS interface.
invented entities (2)
-
Human-AI Multi-Domain Task Force (H-AI MDTF)
-
Multi-Surface Threat (MST)
Cite this review
Pith. "Pith review of Leveraging Trustworthy AI for Automotive Security in Multi-Domain Operations: Towards a Responsive Human-AI Multi-Domain Task Force for Cyber Social Security." pith.science (2026). https://pith.science/paper/3FF5EHQF
@misc{pith2026250721145,
author = {Pith},
title = {Pith review of: Leveraging Trustworthy AI for Automotive Security in Multi-Domain Operations: Towards a Responsive Human-AI Multi-Domain Task Force for Cyber Social Security},
year = {2026},
howpublished = {\url{https://pith.science/paper/3FF5EHQF}},
note = {Machine review of arXiv:2507.21145}
}
read the original abstract
Multi-Domain Operations (MDOs) emphasize cross-domain defense against complex and synergistic threats, with civilian infrastructures like smart cities and Connected Autonomous Vehicles (CAVs) emerging as primary targets. As dual-use assets, CAVs are vulnerable to Multi-Surface Threats (MSTs), particularly from Adversarial Machine Learning (AML) which can simultaneously compromise multiple in-vehicle ML systems (e.g., Intrusion Detection Systems, Traffic Sign Recognition Systems). Therefore, this study investigates how key hyperparameters in Decision Tree-based ensemble models-Random Forest (RF), Gradient Boosting (GB), and Extreme Gradient Boosting (XGB)-affect the time required for a Black-Box AML attack i.e. Zeroth Order Optimization (ZOO). Findings show that parameters like the number of trees or boosting rounds significantly influence attack execution time, with RF and GB being more sensitive than XGB. Adversarial Training (AT) time is also analyzed to assess the attacker's window of opportunity. By optimizing hyperparameters, this research supports Defensive Trustworthy AI (D-TAI) practices within MST scenarios and contributes to the development of resilient ML systems for civilian and military domains, aligned with Cyber Social Security framework in MDOs and Human-AI Multi-Domain Task Forces.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[29]
V. S. Barletta, D. Caivano, C. Catalano, S. D. Vescovo, Black-box adversarial ml attacks on ids and multi-domain impact analysis for threat intelligence in automotive scenarios, in: 2024 IEEE International Workshop on Technologies for Defense and Security (TechDefense), 2024, pp. 132–
work page 2024
-
[1]
F. T. and, Nato’s approach to multi-domain operations: From the perspective of the economics of alliances, Defence and Peace Economics 35 (2024) 281–294. URL: https://doi.org/10.1080/10242694.2023.2235502. doi:10.1080/10242694.2023.2235502. arXiv:https://doi.org/10.1080/10242694.2023.2235502
-
[2]
A. Gilli, M. Gilli, G. G. and, Nato, multi-domain operations and the future of the atlantic alliance, Comparative Strategy 44 (2025) 73–91. URL: https: //doi.org/10.1080/01495933.2024.2445491. doi: 10.1080/01495933.2024.2445491. arXiv:https://doi.org/10.1080/01495933.2024.2445491
- [3]
-
[4]
T. Wójtowicz, D. Król, Multi-domain battle: new doctrine of the united states armed forces, Zeszyty Naukowe Akademii Sztuki Wojennej (2018) 64–78
work page 2018
-
[5]
F.-S. Gady, A. Stronell, Cyber capabilities and multi-domain operations in future high-intensity warfare in 2030, Cyber Threats and NATO 2030: Horizon Scanning and Analysis (2020) 151
work page 2020
-
[6]
V. S. Barletta, D. Caivano, C. Catalano, M. de Gemmis, D. Impedovo, Cyber social security education, in: Extended Reality: International Conference, XR Salento 2024, Lecce, Italy, September 4–7, 2024, Proceedings, Part IV, Springer-Verlag, Berlin, Heidelberg, 2024, p. 240–248. URL: https: //doi.org/10.1007/978-3-031-71713-0_16. doi: 10.1007/978-3-031-71713-0_16
-
[8]
S. G. della Difesa Italiana, The italian defence approach to multi-domain operations (approccio della difesa alle operazioni multidominio), https://www.difesa.it/assets/allegati/31787/2.1defence_ approach_to_mdos.pdf, 2022
work page 2022
Show all 35 references
-
[9]
Campisi, A
T. Campisi, A. Severino, M. A. Al-Rashid, G. Pau, The development of the smart cities in the connected and autonomous vehicles (cavs) era: From mobility patterns to scaling in cities, Infrastructures 6 (2021). URL: https://www.mdpi.com/2412-3811/6/7/100. doi: 10.3390/ infrastr...
2021
-
[10]
Olufowobi, G
H. Olufowobi, G. Bloom, Chapter 16 - connected cars: Automotive cybersecurity and privacy for smart cities, in: D. B. Rawat, K. Z. Ghafoor (Eds.), Smart Cities Cybersecurity and Privacy, Elsevier, 2019, pp. 227–240. doi:https://doi.org/10.1016/B978-0-12-815032-0.00016-0
2019 doi
-
[11]
Morris, G
D. Morris, G. Madzudzo, A. Garcia-Perez, Cybersecurity threats in the auto industry: Tensions in the knowledge environment, Technological Forecasting and Social Change 157 (2020) 120102. URL: https://www.sciencedirect.com/science/article/pii/S0040162520309288. doi:https://doi....
2020
-
[12]
H. Qin, M. Yan, H. Ji, Application of controller area network (can) bus anomaly detection based on time series prediction, Vehicular Communications 27 (2021) 100291. URL: https://www. sciencedirect.com/science/article/pii/S2214209620300620. doi:https://doi.org/10.1016/j. vehco...
2021
-
[13]
Tommasi, C
F. Tommasi, C. Catalano, M. Fornaro, I. Taurino, Mobile session fixation attack in micropayment systems, IEEE Access 7 (2019) 41576–41583. doi: 10.1109/ACCESS.2019.2905219
2019
-
[14]
Rajapaksha, H
S. Rajapaksha, H. Kalutarage, M. O. Al-Kadri, A. Petrovski, G. Madzudzo, M. Cheah, Ai-based intrusion detection systems for in-vehicle networks: A survey, ACM Comput. Surv. 55 (2023). URL: https://doi.org/10.1145/3570954. doi:10.1145/3570954
2023 doi
-
[15]
Sommer, J
F. Sommer, J. Dürrwang, R. Kriesten, Survey and classification of automotive security attacks, Infor- mation 10 (2019). URL: https://www.mdpi.com/2078-2489/10/4/148. doi:10.3390/info10040148
2019 doi
-
[16]
Alfardus, D
A. Alfardus, D. B. Rawat, Intrusion detection system for can bus in-vehicle network based on machine learning algorithms, in: 2021 IEEE 12th Annual Ubiquitous Computing, Electronics & Mo- bile Communication Conference (UEMCON), 2021, pp. 0944–0949. doi:10.1109/UEMCON53757. 202...
2021
-
[17]
Malatras, I
and European Union Agency for Cybersecurity, A. Malatras, I. Agrafiotis, M. Adamczyk, Securing machine learning algorithms, 2021. URL: https://op.europa.eu/publication-detail/-/publication/ c7c844fd-7f1e-11ec-8c40-01aa75ed71a1. doi: doi/10.2824/874249
2021 doi
-
[18]
Aloraini, A
F. Aloraini, A. Javed, O. Rana, Adversarial attacks on intrusion detection systems in in-vehicle networks of connected and autonomous vehicles, Sensors 24 (2024). URL: https://www.mdpi.com/ 1424-8220/24/12/3848. doi:10.3390/s24123848
2024 doi
-
[19]
Longari, F
S. Longari, F. Noseda, M. Carminati, S. Zanero, Evaluating the robustness of automotive intrusion detection systems against evasion attacks, in: Cyber Security, Cryptology, and Machine Learning: 7th International Symposium, CSCML 2023, Be’er Sheva, Israel, June 29–30, 2023, Pr...
2023 doi
-
[20]
B. Wu, Z. Zhu, L. Liu, Q. Liu, Z. He, S. Lyu, Attacks in adversarial machine learning: A systematic survey from the life-cycle perspective, 2024. arXiv:2302.09457
2024 arXiv
-
[21]
Kotyan, A reading survey on adversarial machine learning: Adversarial attacks and their understanding, 2023
S. Kotyan, A reading survey on adversarial machine learning: Adversarial attacks and their understanding, 2023. arXiv:2308.03363
2023 arXiv
-
[22]
E. U. A. for Cybersecurity (ENISA), Artificial intelligence and cybersecurity research, 2023. URL: https://www.enisa.europa.eu/publications/artificial-intelligence-and-cybersecurity-research. doi:10.2824/808362
2023 doi
-
[23]
H.-L. E. G. on AI European Commission, Ethics guidelines for trustworthy ai, 2024. URL: https: //digital-strategy.ec.europa.eu/en/library/ethics-guidelines-trustworthy-ai
2024
-
[24]
N. I. of Standards, Technolgy, Ai fundamental research - security, 2023. URL: https://www.nist. gov/artificial-intelligence/ai-fundamental-research-security
2023
-
[25]
Goellner, M
S. Goellner, M. Tropmann-Frick, B. Brumen, Responsible artificial intelligence: A structured literature review, 2024. URL: https://arxiv.org/abs/2403.06910. arXiv:2403.06910
2024 arXiv
-
[26]
Zenden, H
I. Zenden, H. Wang, A. Iacovazzi, A. Vahidi, R. Blom, S. Raza, On the resilience of machine learning-based ids for automotive networks, in: 2023 IEEE Vehicular Networking Conference (VNC), IEEE, 2023. doi:10.1109/vnc57357.2023.10136285
2023
-
[27]
M. L. Han, B. I. Kwak, H. K. Kim, Anomaly intrusion detection method for vehicular networks based on survival analysis, Vehicular Communications 14 (2018) 52–63. URL: https://www. sciencedirect.com/science/article/pii/S2214209618301189. doi:https://doi.org/10.1016/j. vehcom.20...
2018 doi
-
[28]
H. M. Song, J. Woo, H. K. Kim, In-vehicle network intrusion detection using deep convolutional neural network, Vehicular Communications 21 (2020) 100198
2020
-
[30]
H. Lee, S. H. Jeong, H. K. Kim, Otids: A novel intrusion detection system for in-vehicle network by using remote frame, in: 2017 15th Annual Conference on Privacy, Security and Trust (PST), 2017, pp. 57–5709. doi:10.1109/PST.2017.00017
2017
-
[31]
B. S. Bari, K. Yelamarthi, S. Ghafoor, Intrusion detection in vehicle controller area network (can) bus using machine learning: A comparative performance study, Sensors 23 (2023). doi: 10.3390/ s23073610
2023
-
[32]
Badjie, J
B. Badjie, J. Cecílio, A. Casimiro, Adversarial attacks and countermeasures on image classification- based deep learning models in autonomous driving systems: A systematic review, ACM Computing Surveys 57 (2024) 1–52
2024
-
[33]
Nicolae, M
M. Nicolae, M. Sinn, T. N. Minh, A. Rawat, M. Wistuba, V. Zantedeschi, I. M. Molloy, B. Edwards, Adversarial robustness toolbox v0.2.2, CoRR abs/1807.01069 (2018). arXiv:1807.01069
2018 arXiv
-
[34]
V. S. Barletta, D. Caivano, C. Catalano, M. De Vincentiis, A. Pal, Machine learning for automotive security in technology transfer, in: A. Rocha, H. Adeli, G. Dzemyda, F. Moreira, V. Colla (Eds.), Information Systems and Technologies, Springer Nature Switzerland, Cham, 2024, p...
2024
-
[35]
M. T. Baldassarre, V. S. Barletta, D. Caivano, D. Raguseo, M. Scalera, Teaching cyber security: The hack-space integrated model, in: Italian Conference on Cybersecurity, volume 2315, 2019. URL: https://ceur-ws.org/Vol-2315/paper06.pdf
2019
-
[137]
doi:10.1109/TechDefense63521.2024.10863442
2024
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.