{"id":"3ad8118e-7660-44e7-9ba1-b8f62ac7653d","arxiv_id":"2505.17274","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A survey of 53 CAN intrusion detection systems with a re-implemented benchmark of 19 methods on one real-vehicle dataset, showing that most target classic attacks and struggle with masquerade and diagnostic threats.","lead":"This survey covers 53 intrusion detection systems for the Controller Area Network (CAN) in vehicles, and re-implements 19 of them for comparison on 11 attack datasets collected from a real Toyota Corolla CAN bus. It finds that most existing systems focus on spoofing, fuzzing, and denial-of-service attacks, while diagnostic and stealthier attacks receive less attention, and that many periodicity-based methods fail against masquerade attacks.","discovery_kind":"replication","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline claim that periodic-feature VIDS fail masquerade attacks is not tested: the SPA-3 dataset edits payloads in recorded normal traffic instead of realizing the paper's own masquerade definition.","rationale":"The survey component is genuinely useful: the taxonomy of 18 attack types across 53 VIDS is a substantial contribution, and the qualitative finding that most VIDS target spoofing, fuzzing, and DoS while neglecting diagnostic and stealthy attacks is supported by the collected literature. The evaluation component, however, is where the paper's distinctive claim lives. The reader's weakest assumption was per-dataset threshold tuning (Section VIII.C.1), which is a real flaw: tuning uncertain parameters separately for each attack dataset leaks test information and invalidates the reported accuracy/precision/recall/F1 as estimates of deployment performance. My independent reading points to an even more load-bearing problem at the level of dataset construction: the SPA-3 attack is explicitly built by modifying payloads in normal recorded traffic because pausing the ECU was 'a very big challenge.' That is not the masquerade attack model defined earlier in the paper, where an adversary impersonates a legitimate ECU while the original sender is suspended or bus-off. Therefore, the experiment does not test the central claim that these methods fail against adversaries who 'simulate normal message cycles.' The per-dataset threshold issue would not change the zero-recall result on the edited-payload SPA-3 dataset, since no threshold on message frequency can detect a frequency-preserving edit, but the invalid attack construction directly undermines the generalization from payload edits to masquerade. The paper should either redo the experiment with a true masquerade setup or clearly restrict the claim to stealthy payload tampering. Because the survey and taxonomy remain valuable and the empirical claim is reparable, a conditional acceptance with required revisions is appropriate, matching the reader's verdict.","tokens_in":48171,"tokens_out":4704,"duration_ms":42561,"concrete_test":"Build a true masquerade dataset on the same three-ECU bench: suspend or bus-off the speed ECU (e.g., via a diagnostic programming session or the bus-off mechanism of Section IV.B) and have a second CAN node transmit the victim ID at the normal period with plausible payload dynamics. Fix all VIDS thresholds using only attack-free normal data, as would be done in deployment, and re-run all 19 methods on both this true masquerade dataset and the paper's edited-payload SPA-3 dataset. If the periodicity-based methods still achieve zero recall on the true masquerade dataset, the claim stands; if they achieve nonzero detection due to jitter, physical-layer fingerprint, or payload continuity, the claim must be qualified to the specific attack model actually tested.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central empirical conclusion—that frequency-, entropy-, ID-sequence-, and similarity-based VIDS are 'inherently reliant on the periodicity of CAN messages' and fail against masquerade attacks—depends directly on the SPA-3 experiment in Section VIII.B. The authors write that a real masquerade requires pausing the target ECU 'for a long time, which is a very big challenge,' so they instead 'select the messages with a specific ID and change their payload' and 'modify a piece of normal data every 10 seconds.' That construction is a payload-tampering/stealthy-modification dataset, not a masquerade as defined in Section V.B.3 and Figure 3. A genuine masquerade involves a compromised or additional ECU sending forged frames with the victim ID while the legitimate sender is suspended or driven to bus-off; what must be mimicked is the victim's ID, timing, and possibly physical-layer behavior. Because the experiment never creates that condition, a zero-recall result shows only that these methods miss payload edits in recorded traffic, not that they fail against an adversary simulating normal message cycles. The per-dataset threshold tuning described in Section VIII.C.1 further compromises the quantitative comparisons, but the invalid SPA-3 construction is the more direct threat to the headline claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents a survey of attacks and intrusion detection systems for the Controller Area Network (CAN), together with an empirical evaluation of reimplemented detection methods. The authors classify attacks into a five-category taxonomy (sniffing, spoofing, DoS, fuzzing, diagnostic) with 18 subtypes, survey 53 VIDSs under a feature/technology taxonomy, and reimplement 19 VIDSs on data collected from a real CAN bench built from a 2014 Toyota Corolla. The empirical section reports accuracy, precision, recall, and F1 for each method on 11 attack datasets and concludes that frequency-, entropy-, ID-sequence-, and similarity-based VIDSs rely on CAN message periodicity and fail against masquerade attacks. The paper also discusses practical challenges and future directions for vehicular intrusion detection.","tokens_in":48427,"tokens_out":4854,"duration_ms":40845,"significance":"If its empirical conclusions were fully supported, the paper would be a valuable resource: the survey covers a large body of work, the threat-model and attack taxonomies are useful organizing tools, and the attempt to compare reimplemented methods on a unified dataset goes beyond typical survey practice. The observation that the examined VIDSs concentrate on spoofing, fuzzing, and DoS while neglecting diagnostic and stealthier attacks is well grounded in the collected literature. However, the central empirical claim about masquerade attacks is undermined by the SPA-3 dataset construction, and the per-dataset parameter tuning compromises the quantitative comparison. These issues are local and fixable, but they currently prevent the evaluation from supporting the paper's headline conclusion.","major_comments":[{"comment":"The SPA-3 dataset is not a masquerade attack under the paper's own definition. In Section V.B.3 and Figure 3, a masquerade requires either a compromised or unauthorized sender forging frames with the victim ID while suppressing or replacing the legitimate ECU's transmissions (e.g., via a bus-off attack), so that the attacker simulates the victim's normal message cycle. Section VIII.B instead states that pausing the target ECU 'is a very big challenge' and therefore constructs SPA-3 by selecting messages with a specific ID and changing their payload every 10 seconds. This is a stealthy payload-modification/tampering dataset within recorded normal traffic, not a masquerade as defined. Consequently, the zero precision/recall values for SPA-3 in Figure 6 demonstrate only that these methods miss payload edits while timing and IDs remain unchanged; they do not demonstrate failure against an adversary that simulates the victim ECU's normal transmission pattern. The Brief Discussion and abstract conclusions that frequency-, entropy-, ID-sequence-, and similarity-based methods fail against masquerade attacks are therefore not supported by the reported experiment. The authors should either implement a genuine masquerade scenario (e.g., bus-off followed by forged injection, or a suspended sender with attacker frames matching the victim's ID/timing) or rename SPA-3 as a stealthy payload-modification attack and temper the corresponding claims.","section":"§VIII.B, §V.B.3, Fig. 6"},{"comment":"The evaluation tunes uncertain parameters separately for each attack dataset 'to achieve the best effectiveness of these VIDSs.' For example, the entropy threshold parameter k in [70] is adjusted per dataset. This means the reported accuracy, precision, recall, and F1 values in Figure 6 are per-dataset best-case results obtained by fitting thresholds to each test set, not results of a fixed-configuration benchmark. As a consequence, the comparative statements among methods and attacks may reflect favorable threshold choices rather than general detection ability. The authors should fix uncertain parameters on a training or validation split (or at least report both fixed and per-dataset-tuned settings), and should clearly label the current numbers as upper bounds under per-dataset tuning. This issue is load-bearing for the quantitative comparison.","section":"§VIII.C.1, Fig. 6"},{"comment":"The reproduced set includes only parameters-monitoring-based and payload-based methods; fingerprint-based VIDSs such as clock-skew, voltage, and bit-time methods are excluded because they require special data collection equipment. The conclusion in the Brief Discussion and abstract should therefore be scoped explicitly to the evaluated method families. As currently written, statements such as 'these methods' can be read as applying to all anomaly-based VIDS, whereas fingerprint-based methods are specifically designed to detect masquerade-style impersonation and were not evaluated. Adding an explicit scope sentence in the abstract and in Section VIII.D would prevent overgeneralization.","section":"§VIII.A, §VIII.D"}],"minor_comments":[{"comment":"The attack labeled 'Disorderly Control attack (SPA-2)' is mislabeled: SPA-2 is Fabrication in the taxonomy of Section V.B and Figure 3, while Disorderly Control is FUA-1. The same dataset file is referred to as FUA-1 in Table V, so the text and table are inconsistent.","section":"§VIII.B"},{"comment":"The heading 'Inconsitence Defeination of Attack Models' contains typos; it should read 'Inconsistency in Definition of Attack Models.'","section":"§VIII.C.2"},{"comment":"The subfigures in Figure 6 are visually dense, with four metrics plotted on the same y-axis labeled 'Probability.' A tabular listing of the numerical values or separate, larger subfigures would make the results substantially more readable and verifiable.","section":"Fig. 6"},{"comment":"The authors' own SAID system (reference [135]) appears in Table IV but is not described in the main survey taxonomy in Section VI or in Table III. If it is included as a surveyed system, it should be described like the others; if it is included only for comparison, the paper should state this explicitly and disclose the self-citation.","section":"Table IV, Table III"},{"comment":"The evaluation is based on a bench with only three ECUs and 23 CAN IDs from a single vehicle model. The paper should state more prominently that the empirical results are a single-platform case study and that generalization to other vehicles and network architectures remains untested.","section":"§VIII.B, Table V"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within scope for a venue interested in automotive or applied security. The SPA-3 construction is the main obstacle: it is a payload-tampering dataset presented as a masquerade attack, and it directly supports the paper's headline empirical claim. The per-dataset tuning is also a substantive methodological concern. Both are fixable within the manuscript's scope, so I recommend major revision rather than rejection. I also note that the authors' own SAID system appears in Table IV without a corresponding description in the survey taxonomy; this deserves a transparency note, though I do not see evidence of deliberate misrepresentation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a genuinely useful survey and a rare attempt at head-to-head reproduction of 19 VIDS on one real-vehicle dataset. But the headline empirical claim—that frequency/entropy/ID-sequence/similarity methods fail masquerade attacks—is not actually tested. The SPA-3 dataset in Section VIII.B is payload tampering, not a masquerade by the paper's own definition in Section V.B.3. The authors admit they couldn't suspend the ECU, so they changed payloads of selected messages. That shows these methods miss some payload edits, not that they fail against an adversary who simulates the victim's message cycle. The qualitative survey finding—that the field concentrates on spoofing/fuzzing/DoS while ignoring diagnostic and stealthy threats—is well supported and is the paper's real contribution.\n\nThe taxonomy is more detailed than prior surveys: 18 attack types, six spoofing variants, and a clear threat-model framework. The evaluation is also a serious effort: reimplementing 19 methods and running them on the same bench is more than any prior survey did. I believe the authors when they say the comparison took work.\n\nNow the soft spots. First, the SPA-3 problem is load-bearing for the paper's central conclusion. Precision/recall/F1 of zero against SPA-3 are cited throughout the discussion as evidence of inherent periodicity-dependence. Since SPA-3 is not a masquerade, those numbers mean little. Second, Section VIII.C.1 states thresholds were tuned per dataset to achieve \"the best effectiveness.\" That turns the benchmark into a per-method, per-attack fitting exercise; the reported scores are upper bounds, not out-of-the-box performance. Third, the bench is tiny—three ECUs, 23 IDs, one vehicle model—and no code, data, or error bars are released, so others can't verify or extend. There are also small slips (e.g., \"Disorderly Control attack (SPA-2)\" is labeled with a spoofing code; it should be FUA-1), but those are minor. Finally, the paper's suggestion that payload-based methods defend against masquerade is unsupported because SPA-3 is exactly the case a payload model should catch—real masquerade would simulate payload too.\n\nBottom line: the survey and taxonomy deserve publication after revision, and a serious referee should be assigned. The evaluation needs either a major rework—fixed parameters, a genuine masquerade setup, released artifacts, variance reporting—or should be reframed as a less conclusive pilot.","headline":"Useful survey and a brave but compromised benchmark: the \"masquerade\" experiment does not actually implement the paper's own masquerade definition.","tokens_in":48980,"tokens_out":2815,"would_cite":true,"duration_ms":32311,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Most CAN bus intrusion detectors are periodicity-dependent and fail against masquerade attacks that preserve normal message timing, according to a unified re-evaluation of 19 detection methods on real vehicle traffic.","keywords":["controller area network","vehicular intrusion detection","CAN bus security","masquerade attack","periodicity assumption","anomaly detection evaluation","automotive cybersecurity","diagnostic attack"],"falsifier":"Re-run the 19 detectors on a second vehicle's CAN traffic or on a public CAN intrusion dataset, fixing all thresholds after training on normal traffic only, and check whether the masquerade-attack F1 scores remain near zero; any frequency-based detector that then reports high F1 would falsify the claim that periodicity-based methods are inherently blind to masquerade attacks.","tokens_in":47955,"feed_emoji":"🚗","tokens_out":7360,"duration_ms":50891,"temperature":0.7,"pith_summary":"This paper tries to establish that the field of vehicular intrusion detection for the Controller Area Network has been aimed at the wrong target: most proposed VIDS concentrate on classic spoofing, fuzzing, and denial-of-service attacks, while realistic and newer threats—especially diagnostic attacks and masquerade attacks that imitate a legitimate ECU—are underrepresented. To back this, the authors survey 34 attack studies, classify 53 VIDS by the feature they monitor, and re-implement 19 detection methods that can be run on a common dataset collected from a real CAN bus. The unified evaluation is the load-bearing part: it shows that frequency-based, information-entropy-based, ID-sequence-based, and similarity-based detectors all lean on the periodicity of CAN messages and report near-zero detection on masquerade attacks that keep the victim's message timing intact. The paper concludes that payload-based methods offer the only reimplemented route to masquerade detection but are themselves limited by the difficulty of modeling normal payloads, and that future VIDS should be tested against the broader attack taxonomy.","feed_headline":"19 CAN intrusion detectors miss masquerade attacks","feed_subtitle":"A unified evaluation on real CAN traffic shows periodicity-based vehicle defenses cannot spot forged messages that keep normal timing.","key_machinery":"The argument is carried by a controlled re-implementation experiment rather than by a single new algorithm. The authors select 19 detection methods from 17 papers spanning the feature families they taxonomize (frequency, information entropy, ID sequence, similarity, payload), and run them on one testbed built from the CAN hardware of a 2014 Toyota Corolla with three ECUs and 23 message IDs, using 11 datasets that cover replay, fabrication, masquerade, DoS, fuzzing, and diagnostic attacks. The taxonomy of attacks (sniffing, spoofing with six subtypes, DoS with five subtypes, fuzzing, and diagnostic attacks) and the classification of VIDS by monitored feature are what make the comparison interpretable: each family's detection principle is tied to a specific assumption about CAN traffic, and the experiment shows which assumptions break.","core_discovery":"The central claim is that periodicity is the hidden assumption of the most common CAN intrusion detectors, and that a same-dataset comparison exposes it. In the authors' experiments, the frequency-based, entropy-based, ID-sequence-based, and similarity-based methods detect high-rate injection, DoS, and fuzzing attacks well, but they fail on the SPA-3 masquerade attack, where forged messages replace normal ones while preserving the time interval from the preceding message; several detectors get precision, recall, and F1 of zero on that attack. The same methods also cannot separate normal diagnostic messages from malicious diagnostic traffic, because diagnostic messages do not follow a stable periodic pattern. Payload-based methods, which look for changes in the data field rather than in timing, are the family that can in principle catch masquerade attacks, but the paper reports that building a reliable model of normal CAN payloads is hard, so their detection rates remain low.","pith_inferences":["Beyond the paper: because thresholds were tuned per attack dataset, the absolute accuracy and F1 numbers likely overstate the deployable performance of every method; the qualitative ranking of families may hold, but the magnitudes should be read as upper bounds.","Beyond the paper: the same evaluation protocol could be extended to fingerprint-based VIDS (clock skew, voltage, bit time) if a shared physical-layer dataset were captured, which would test whether the periodicity-independent families actually close the masquerade gap the paper identifies.","Beyond the paper: a natural next experiment is a combined detector that fuses a periodicity check with a payload or physical-layer fingerprint; the survey's analysis predicts such a hybrid would be the minimal design able to cover both high-rate injection and masquerade attacks.","Beyond the paper: the diagnostic-attack results imply that any VIDS deployed on vehicles that allow OBD-II diagnostics should be validated on diagnostic sessions, not just on normal periodic traffic."],"forward_implications":["Frequency-, entropy-, ID-sequence-, and similarity-based VIDS can catch attacks that change message counts, IDs, or regularity, but they will miss a masquerade attack that preserves the victim's message cycle.","Diagnostic attacks are effectively outside the detection scope of the reimplemented periodicity-based VIDS, and even benign diagnostic traffic is frequently misclassified as normal.","Payload-based detection is the only reimplemented family with a plausible path to catching masquerade attacks, but its practical ceiling is set by how well normal payloads can be modeled.","The survey's attack taxonomy gives future VIDS a checklist: designs that target only spoofing, fuzzing, and DoS are leaving diagnostic, cloaking, and voltage-corruption threats uncovered."],"supporting_citations":[{"why":"Supplies the clock-skew fingerprinting VIDS included in the unified evaluation and the two-ECU masquerade attack model.","marker":"[20]"},{"why":"The time-interval frequency-based VIDS whose near-zero masquerade detection illustrates the periodicity failure.","marker":"[66]"},{"why":"The entropy-based detector (two variants) re-implemented in the evaluation; central to the entropy failure claim.","marker":"[70]"},{"why":"The ID-sequence anomaly detector re-implemented and shown to rely on periodic ID order.","marker":"[120]"},{"why":"The sliding-window similarity detector re-implemented for the DoS and masquerade comparison.","marker":"[122]"},{"why":"The payload Hamming-distance method that represents the only reimplemented family with a route to masquerade detection.","marker":"[25]"},{"why":"The voltage-corruption attack used to argue that modern sophisticated attacks evade existing VIDS and are outside most threat models.","marker":"[38]"},{"why":"The experimental vehicle security study that introduced the classic spoofing, fuzzing, and DoS attack classes most VIDS still target.","marker":"[48]"}],"fun_headline_variants":["Masquerade attacks slip past timing-based CAN intrusion detectors","Periodicity assumption blinds CAN intrusion detection to forged messages","CAN IDS rely on timing that masquerade attacks preserve","Timing-preserving masquerade attacks defeat common CAN IDS"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes that tuning each detector's uncertain parameters separately on each attack dataset still measures real-world detection ability; if those thresholds are overfit to the test attacks, the reported scores flatter the methods and the comparative conclusions may not generalize to other vehicles or traffic conditions.","fun_headline_variants_meta":{"raw":{"variants":["Masquerade attacks slip past timing-based CAN intrusion detectors","Periodicity assumption blinds CAN intrusion detection to forged messages","CAN IDS rely on timing that masquerade attacks preserve","Timing-preserving masquerade attacks defeat common CAN IDS"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000917,"raw_usage":{"total_tokens":3926,"prompt_tokens":924,"completion_tokens":3002,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":540,"completion_tokens_details":{"reasoning_tokens":2933}},"tokens_in":540,"tokens_out":3002,"duration_ms":16854,"temperature":1.0,"reasoning_tokens":2933,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:49:15.030995+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the 19 detectors on a second vehicle's CAN traffic or on a public CAN intrusion dataset, fixing all thresholds after training on normal traffic only, and check whether the masquerade-attack F1 scores remain near zero; any frequency-based detector that then reports high F1 would falsify the claim that periodicity-based methods are inherently blind to masquerade attacks.","supporting_citations":[{"cited_title":"Anomaly detection of can bus messages through analysis of id sequences,","cited_arxiv_id":null,"evidence_quote":"The ID-sequence anomaly detector re-implemented and shown to rely on periodic ID order."},{"cited_title":"Normal and malicious sliding windows similarity analysis method for fast and accurate ids against dos attacks on in-vehicle networks,","cited_arxiv_id":null,"evidence_quote":"The sliding-window similarity detector re-implemented for the DoS and masquerade comparison."}],"review_version":1}