{"id":"34a44cc2-a8a8-4d5f-9a21-39856923d0ca","arxiv_id":"2507.21145","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Larger Random Forest and Gradient Boosting ensembles increase the time needed for a ZOO black-box attack on a CAN bus IDS, while XGBoost shows no clear relationship.","lead":"This paper measures how the number of trees in Random Forest and the number of boosting rounds in Gradient Boosting affect the time an attacker needs to fool a CAN bus intrusion detection system with a black-box adversarial attack. It finds larger ensembles slow the attack for RF and GB but not clearly for XGB, and argues this can buy defenders time in multi-domain operations.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"RQ2's 'window of opportunity' claim is unsupported: attack times are measured only on the pre-adversarial-training model, so the defender/attacker timing comparison never tests the actual hardened model.","rationale":"The reader's weakest assumption concerns representativeness of the B' subset, which would affect generalization of the RQ1 timing measurements. That is a valid concern, but the more immediate logical gap is RQ2: the paper never measures attack time against the adversarially trained model. The strongest claim explicitly frames the contribution as intrinsic defense plus AT reducing the attacker's window, and the reported data cannot support that framing because the pre-AT attack time is compared only with AT training time. This is an internal incompleteness, not a disagreement with the field's consensus. It is also explicitly acknowledged in the future-work section, which strengthens the case that the current conclusion is premature. A single targeted experiment--measuring post-AT attack time for at least one RF and one GB configuration--would settle whether RQ2 holds. If the post-AT times remain favorable, the conditional accept stands; if not, RQ2 should be downgraded and the paper's contribution narrowed to RQ1. The reader's verdict of CONDITIONAL is therefore appropriate, and this stress-test does not move it.","tokens_in":11210,"tokens_out":6555,"duration_ms":73539,"concrete_test":"Use the same ZOO configuration (learning_rate=0.1, max_iter=50, variable_h=0.2) to attack the adversarially trained Model_A+B+B' for at least the settings highlighted in the paper (RF with ~81 trees, GB with ~80 rounds), generating 92,270 adversarial examples from C' or another held-out split not used in AT. Record wall-clock time and compare with Figures 2 and 3. If post-AT attack time is not larger than the corresponding pre-AT attack time plus AT time, RQ2 does not hold; if it is larger, the missing measurement can be supplied and would support the conditional verdict.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central RQ2 conclusion (Section 4.1, Figures 2-6) is that combining adversarial training with hyperparameter tuning shrinks the attacker's window of opportunity. However, the attack times in Figures 2-4 are measured on Model_A, the IDS trained only on split A, while adversarial training produces a different model, Model_A+B+B' (Section 3.2, step 3). The time an attacker would need against that hardened model is never measured; Section 5 explicitly lists 'attack times following the application of the countermeasure' as future work. Comparing AT training time with pre-AT attack time cannot establish a reduced window: the defender may train faster than the original attack, but the attacker's cost against the AT model could be higher, lower, or unchanged, and only the post-AT number determines the operational window. Thus RQ2, and the D-TAI/MDTF impact narrative built on it, rests on a missing measurement rather than on the data actually reported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies whether the number of estimators/boosting rounds in Random Forest (RF), Gradient Boosting (GB), and Extreme Gradient Boosting (XGB) affects the time required to generate adversarial examples with the ZOO black-box attack against an ML-based CAN bus intrusion detection system (IDS). Using the OTIDS dataset and a pipeline inherited from the authors' previous work, the paper measures attack-generation time on a subset B' and adversarial-training (AT) time on the combined training set. It reports a positive linear trend for RF and GB, a non-trend for XGB, and concludes that careful hyperparameter selection combined with AT can reduce the attacker's 'window of opportunity.' It then qualitatively maps this conclusion to the Detection-Response-Prevention axes of the CSS-MDO framework to support educational claims about Human-AI Multi-Domain Task Forces.","tokens_in":11448,"tokens_out":2956,"duration_ms":36070,"significance":"If the empirical claims held, the paper would provide a practically useful, low-cost defensive lever: choosing ensemble sizes can increase the computational cost of black-box adversarial examples without altering the deployed model family. The direct timing measurements, the use of a standard attack library (ART), and the concrete automotive threat model are strengths. However, the significance is substantially limited by the fact that the central operational claim (RQ2) relies on comparing AT training time with attack time measured only on the pre-AT model, and the statistical evidence for the RQ1 trends is thin (no error bars, no goodness-of-fit values, no repetitions). The qualitative RQ3 analysis is appropriately framed as qualitative, but it inherits the weaknesses of the underlying quantitative claims.","major_comments":[{"comment":"The regression lines in Figures 2–4 are presented with coefficients but no goodness-of-fit statistics, confidence intervals, or repeated trials. The claim that RF and GB show a 'clear linear relationship' while XGB does not is not statistically substantiated. RQ1's affirmative/negative answers hinge on this distinction; please report at least R² or adjusted R², and ideally per-configuration error bars or repeated measurements, for every regression.","section":"§4.1, Figures 2–4"},{"comment":"RQ2's affirmative answer is unsupported because attack time is measured only on Model_A (the pre-adversarial-training model), while the defender's AT timing is measured on the hardened Model_A+B+B'. The attacker's time against the AT model is never measured; Section 5 explicitly lists 'attack times following the application of the countermeasure' as future work. Without that measurement, comparing pre-AT attack time with AT training time cannot establish a reduced operational window. This is a load-bearing gap for the paper's D-TAI/MDTF impact narrative.","section":"§4.1, RQ2 (Figures 2–6)"},{"comment":"The empirical timing study is conducted on subset B' under the stated assumption that B' 'is consistent with C''. No evidence is provided for this assumption: no distributional comparison, no cross-validation on other splits, and no report of whether the timing relationship changes on C'. Since all RQ1 conclusions and the extrapolated attack times (e.g., ~31 hours for 81 RF trees) are derived solely from B', the lack of validation weakens the generality of the results. Please quantify representativeness or repeat the timing study on at least one held-out split.","section":"§3.3"},{"comment":"The statement that AT time is 'significantly less than the time provided to the H-AI MDTF to detect the intrusion in the CAN network' introduces a detection-time quantity that is never defined or measured in the paper. Without a concrete definition and measurement of intrusion-detection latency, this comparison is unsupported and should be removed or explicitly identified as a hypothesis.","section":"§4.1, final paragraph"}],"minor_comments":[{"comment":"The paper contains several typos and grammatical errors, including 'realted', 'prelaborated', 'demage', 'coutermeasure', 'consedered', and 'examinated'; a careful proofreading pass is needed.","section":"§5"},{"comment":"The acronym 'TT' is introduced as 'Technology Transfer' but the models RF/GB/XGB are ordinary supervised classifiers; the connection to technology transfer is not explained and may confuse readers.","section":"§1 and §2.1"},{"comment":"The figure captions and axis labels use 'Estimated Time' for attack and training time, but the estimation method is not described; clarifying whether these are single-run measurements, averages, or fitted values would improve transparency.","section":"§4.1, Figure captions"},{"comment":"The impact ratings ('Very High', 'High') and motivations are self-assessed with no defined rating scale or assessment procedure; the table should be labeled as an illustrative qualitative assessment rather than an empirical evaluation result.","section":"§4.2, Table 1"}],"recommendation":"major_revision","confidential_remarks":"The paper matches the workshop's interdisciplinary scope, but the technical core is thin: RQ2's main claim is missing its key measurement, and the statistical analysis of RQ1 needs strengthening. The authors should be given the opportunity to add the post-AT attack-time experiments and report basic goodness-of-fit statistics. If those cannot be provided, the conclusions should be scaled back to a descriptive timing study."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The genuinely new thing here is the measurement of ZOO attack time as a function of ensemble-size hyperparameters for RF, GB, and XGB on a CAN bus IDS task. The RQ1 result—that more trees in RF and more boosting rounds in GB slow the attack, while XGB shows no such trend—is a plausible and directly measured effect. If it holds up, it gives IDS designers a cheap lever: increase ensemble size to raise the attacker's cost. That is worth knowing, even if it is an incremental extension of the authors' own prior work [29] rather than a new mechanism.\n\nThe paper is honest about its scope. It explicitly defers attack-time measurement after adversarial training to future work, which is exactly where the main problem sits. RQ2 claims that combining hyperparameter tuning with adversarial training shrinks the attacker's window of opportunity, but the attack times in Figures 2–4 are all measured against Model_A, the pre-AT model. The hardened model, Model_A+B+B', is never attacked. Comparing AT training time with pre-AT attack time cannot tell you anything about the attacker's actual cost against the deployed, hardened model. That cost could be higher, lower, or unchanged. So RQ2, and the entire D-TAI/MDTF narrative built on it, is unsupported by the data actually reported. The stress-test note is right on target.\n\nSmaller soft spots, in proportion: the B' representativeness assumption is load-bearing and unvalidated; there are no error bars, repetitions, or goodness-of-fit statistics on the regressions; no code or data release; and Table 1's impact ratings are self-assessed. None of these are fatal to the RQ1 timing observation, but they cap the paper's strength. The citation pattern looks fine—self-citation is appropriate because the pipeline and prior attack results are genuinely their own.\n\nWho is this for? A reader who wants a quick empirical hint about ensemble-size effects on black-box attack time in this specific automotive IDS setting. It is not a definitive study, and the RQ2 conclusion should not be cited. But the RQ1 measurement is a legitimate, falsifiable data point. I would send it to a serious referee—it is the kind of incremental empirical work that deserves scrutiny rather than a desk reject—and the referee should be instructed to focus on the missing post-AT attack-time measurement and the B' assumption.","headline":"A straightforward empirical extension of the authors' own prior ZOO-attack study: the RQ1 timing measurements are credible and potentially useful, but the RQ2 window-of-opportunity claim rests on a missing measurement of post-adversarial-training attack time.","tokens_in":11939,"tokens_out":638,"would_cite":false,"duration_ms":7955,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that Random Forest and Gradient Boosting can be configured so that the ZOO black-box attack takes much longer to succeed, while XGB cannot.","keywords":["Multi-Domain Operations","Trustworthy AI","Automotive Security","Adversarial Machine Learning","Zeroth Order Optimization","CAN bus intrusion detection","Random Forest","Gradient Boosting"],"falsifier":"Run the same ZOO attack-time experiment with the same three models but on a different split of OTIDS or on another CAN-bus dataset, such as the Survival or Car Hacking datasets; if attack time does not increase with tree or boosting-round counts there, the intrinsic-defense claim fails. A second check is to see whether XGB shows a linear increase under different ZOO learning-rate settings.","tokens_in":11039,"feed_emoji":"🛡️","tokens_out":5517,"duration_ms":53909,"temperature":0.7,"pith_summary":"This paper argues that the configuration of decision-tree ensemble models used in CAN-bus intrusion detection systems can act as a built-in defense against black-box adversarial machine learning. Specifically, it claims that increasing the number of trees in Random Forest or boosting rounds in Gradient Boosting substantially increases the time an attacker needs to generate adversarial examples with the ZOO attack, while XGB does not show the same effect. If true, a vehicle security operator can slow down an attacker simply by choosing more expensive ensemble settings, and combine this with adversarial training to widen the detection-and-response window. The paper also maps this practice onto the Cyber Social Security framework for multi-domain operations, arguing that it should be taught to future Human-AI task forces.","feed_headline":"Tuning tree counts can slow CAN-bus AI attacks by hours","feed_subtitle":"More ensemble members stretch adversarial example generation, widening the defender's response window.","key_machinery":"The load-bearing mechanism is the dependence of ZOO attack runtime on model ensemble size: each adversarial example requires many numerical gradient-estimation queries through the victim model, so a model with more trees or boosting rounds takes longer to evaluate. This turns a hyperparameter (number of estimators) into an intrinsic defense, measured by the slope of attack time versus estimator count under linear regression.","core_discovery":"The paper reports that, for a pure black-box ZOO attack against supervised machine-learning-based intrusion detection on CAN bus frames, the time needed to produce 92,270 adversarial examples grows linearly with the number of bagging trees in Random Forest and with the number of boosting rounds in Gradient Boosting, while Extreme Gradient Boosting shows no such consistent relationship. On this basis the paper treats tree-count and boosting-round settings as intrinsic defensive mechanisms, and shows that adversarial training time is comparatively small, so a defender can combine these settings with adversarial training to widen the window in which a Human-AI task force can detect an intrusion.","pith_inferences":["We infer that the same hyperparameter-slowing effect may hold for other gradient-estimation black-box attacks, but the paper only tests ZOO, so this is an extension rather than a result.","A natural testable extension is to check whether an attacker can compensate by raising ZOO's learning rate or reducing its iteration count, which would reveal whether the defense is robust against adaptive attackers.","Because the measurements rely on a single dataset subset, replicating the experiment on a different CAN-bus dataset would show whether the qualitative ordering RF and GB sensitive, XGB not persists in other traffic distributions."],"forward_implications":["For Random Forest, the reported linear fit implies each additional bagging tree adds roughly 21 minutes of ZOO attack time for the tested adversarial-example set, so increasing tree count is a tunable way to slow an attacker.","For Gradient Boosting, each extra boosting round adds about 25 seconds of attack time, giving a weaker but still measurable slowdown.","For Extreme Gradient Boosting, boosting rounds do not produce a consistent linear increase in attack time, so XGB does not offer the same intrinsic-defense benefit.","Adversarial training time stays on the order of minutes while attack time is on the order of hours, so the defender's countermeasure is much cheaper than the attacker's effort, and the paper recommends Random Forest as the more defensive choice."],"supporting_citations":[{"why":"Supplies the ZOO attack pipeline, victim model definitions, and the prior accuracy-degradation result that this study extends to hyperparameter timing.","marker":"[29]"},{"why":"Supplies the OTIDS CAN-bus dataset from which subsets A, B, and C are derived and on which all timing measurements are made.","marker":"[30]"},{"why":"Defines the dataset preprocessing pipeline that produces the final split used for training and adversarial-example generation.","marker":"[31]"},{"why":"Implements the ZOO attack used to generate adversarial examples and measure attack time.","marker":"[33]"},{"why":"Defines the CSS-MDO framework axes that the qualitative impact analysis maps onto for RQ3.","marker":"[7]"}],"fun_headline_variants":["Tree count tweaks stretch CAN-bus attack time","Ensemble size widens defender's response window","Hyperparameter tuning buys hours against CAN attacks","RF and GB slow black-box attacks; XGB stable","Boosting rounds lengthen adversarial example generation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The timing measurements are taken only on one subset (B') of one CAN-bus dataset and are assumed to represent the full data distribution, so the reported slowdown from tuning hyperparameters may not generalize to other traffic or other datasets.","fun_headline_variants_meta":{"raw":{"variants":["Tree count tweaks stretch CAN-bus attack time","Ensemble size widens defender's response window","Hyperparameter tuning buys hours against CAN attacks","RF and GB slow black-box attacks; XGB stable","Boosting rounds lengthen adversarial example generation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000306,"raw_usage":{"total_tokens":1734,"prompt_tokens":904,"completion_tokens":830,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":520,"completion_tokens_details":{"reasoning_tokens":758}},"tokens_in":520,"tokens_out":830,"duration_ms":9897,"temperature":1.0,"reasoning_tokens":758,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T14:47:40.208049+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same ZOO attack-time experiment with the same three models but on a different split of OTIDS or on another CAN-bus dataset, such as the Survival or Car Hacking datasets; if attack time does not increase with tree or boosting-round counts there, the intrinsic-defense claim fails. A second check is to see whether XGB shows a linear increase under different ZOO learning-rate settings.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the ZOO attack pipeline, victim model definitions, and the prior accuracy-degradation result that this study extends to hyperparameter timing."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the OTIDS CAN-bus dataset from which subsets A, B, and C are derived and on which all timing measurements are made."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the dataset preprocessing pipeline that produces the final split used for training and adversarial-example generation."}],"review_version":1}