{"id":"a534f7a8-4c29-4c49-8ac0-01b5ccb0782c","arxiv_id":"2412.02539","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"Graph-based models on UAV CAN bus traffic beat an LSTM in some scenarios, but the transfer ID feature contradicts the protocol-independent claim and several attacks go undetected.","lead":"Researchers turned drone CAN bus traffic into graphs and tested whether graph-based machine learning models can detect injected cyberattacks. The approach shows high accuracy in some scenarios, but the protocol-independent claim depends on a decoded transfer ID and several attack scenarios are missed.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The protocol-independence claim is contradicted by the method: Section VI-A/B use Transfer ID, a UAVCAN-decoded payload field, for graph self-loops, so the claimed comparison without decoded features is not actually performed.","rationale":"Reading in good faith, the empirical study may still show that graph models outperform an LSTM in some scenarios. But the headline contribution is protocol independence, and that claim rests on the assumption that Transfer ID is a structural feature rather than a protocol-decoded payload field. The paper itself contradicts this assumption in Sections VI-A and VI-B. This is not a matter of disagreeing with consensus; it is an internal inconsistency in the method as described. The reported accuracy values are also misleading because class-imbalanced scenarios 2 and 4 show F1 near 0.5, so higher detection performance is not supported by all metrics. The absence of code and full configuration means the remaining empirical findings cannot be independently checked. This confirms the reader's REJECT verdict; I would not soften it without a corrected ablation that removes Transfer ID and shows no performance loss.","tokens_in":17906,"tokens_out":6205,"duration_ms":63223,"concrete_test":"Using the public HCRL UAVCAN dataset [27], rerun the complete pipeline without the Transfer ID feature: build edges from consecutive CAN IDs exactly as in Equation 1 but drop the transfer-ID constraint (for example, treat every consecutive identical CAN ID as a new message, or introduce no self-loops at all). Keep window size, density window, PageRank, class weights, and model configurations identical to Sections VI and VII. Recompute all ten scenarios for GraphSAGE, graph transformer, GCNN, and GAT, reporting precision, recall, and F1 rather than accuracy alone. If F1 drops materially in scenarios involving multi-frame UAVCAN messages, or if the ranking versus the LSTM baseline changes, the method is dependent on the decoded Transfer ID and the protocol-independence claim is false. If all metrics remain equivalent, the claim could be revised, but it is currently unsubstantiated.","verdict_should_be":"REJECT","load_bearing_attack":"The paper's central claim is that the graph-based IDS achieves higher detection performance without using any decoded features based on the UAVCAN protocol and is therefore protocol-independent. The described pipeline does not support this claim. Section VI-A says that during graph construction one feature, transfer ID, is used, which defines self-loops in graphs, and Section VI-B's self-loop constraint uses Transfer ID_ti != Transfer ID_ti+1 to decide whether consecutive identical CAN IDs are new messages. Transfer ID is not a raw structural property of the CAN bus; it is encoded in the payload tail byte and must be extracted by decoding the UAVCAN data field (Section VI-A, Figure 5). The HCRL dataset [27] ships already-decoded messages, so the statement that no further protocol-level decoding was performed does not remove the dependency: a deployed protocol-independent IDS would not have a pre-decoded Transfer ID column. Removing Transfer ID would alter the graph construction for multi-frame messages, so the reported comparison against a payload-decoding LSTM does not demonstrate protocol independence. The performance claim is also internally weakened by Tables II-V: in Scenarios 2 and 4, all graph models report precision/recall/F1 near 0.5 while accuracy exceeds 0.97, indicating near-random attack detection under class imbalance. As written, the central claim fails.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes a graph-based intrusion detection system for UAV CAN bus traffic using the UAVCAN protocol. The authors decode CAN messages from the HCRL dataset, convert streams of CAN IDs into time-windowed graphs with timestamp-based edge weights and Transfer-ID-dependent self-loops, compute PageRank and density features, and train GCNN, GAT, GraphSAGE, and graph-transformer classifiers. They compare against a single-layer LSTM baseline and claim that the graph models outperform it while being protocol-independent and using no UAVCAN-decoded payload features. The paper concludes that graph methods capture CAN communication structure better than sequence models.","tokens_in":18162,"tokens_out":7860,"duration_ms":76199,"significance":"The problem is timely, and the idea of representing CAN traffic as temporal communication graphs is a reasonable direction; the comparison across ten attack scenarios is also useful. However, the paper's central contribution, protocol-independent detection without decoded payload features, is contradicted by the method's use of Transfer ID, a decoded UAVCAN payload field. In addition, the reported results in several scenarios show F1/precision/recall around 0.5, which indicates near-random attack detection despite high accuracy, and the comparison with the LSTM is not a controlled experiment. If the protocol-independence claim were properly supported, the work would be of practical interest; as written, the evidence does not support it.","major_comments":[{"comment":"The protocol-independence claim is contradicted by the graph construction. Section VI-A states that \"one feature, transfer ID, is used, which defines self-loops in graphs,\" and Figure 5 shows that the transfer ID is embedded in the payload tail byte of UAVCAN messages. Section VI-B then uses the constraint Transfer ID_ti != Transfer ID_ti+1 in Eq. (1) to decide whether consecutive identical CAN IDs represent a new multi-frame message. Transfer ID is therefore a decoded protocol payload field, not a raw structural property of the CAN bus. The abstract and contributions claim that the graph models perform best \"without using any decoded features based on the UAVCAN protocol,\" but no such variant is evaluated. If Transfer ID is removed, the self-loop structure changes and the reported performance may change; no results are given for that protocol-independent version.","section":"§VI-A, §VI-B, Eq. (1)"},{"comment":"Several reported scenarios do not indicate successful detection. In Table II, Scenario 2 shows precision 0.511, recall 0.740, F1 0.516, and Scenario 4 shows precision 0.500, recall 0.499, F1 0.499; Tables III–V report similar values near 0.5 for Scenarios 2 and 4. Since F1 = 0.5 with accuracy above 0.97 is the signature of a classifier that essentially predicts the majority class, the text's claims of \"high precision, recall, and F1-score\" (Section VII) are not supported for these scenarios. This matters because Scenarios 2 and 4 are not edge cases; they are two of the four single-attack flooding and fuzzy scenarios.","section":"Tables II–V"},{"comment":"The claim that \"all our graph-based models perform better\" than the LSTM is not supported by the reported numbers. In Table VI, GAT accuracy in Scenario 6 is 0.697, below the LSTM's 0.706, and the conclusion's statement that GB-ML models show improvements of 1% to 29% \"across all attack scenarios\" is therefore inaccurate. Moreover, Table VI reports only accuracy, while Section VIII claims improvements \"in terms of precision, recall, and F1 scores.\" The paper does not provide the additional tables needed to support that claim.","section":"Table VI"},{"comment":"The evaluation lacks the experimental detail needed to compare models. There are no train/test splits, no repeated runs, no confidence intervals or error bars, and no hyperparameters (e.g., GNN layers, hidden dimensions, learning rate, dropout, number of epochs, or the actual class weights used to address imbalance). The LSTM baseline is imported from [59] without specifying whether it uses the same splits, features, and attack scenarios. Under these conditions, the accuracy differences in Table VI cannot be distinguished from noise.","section":"§VII, Tables II–VI"}],"minor_comments":[{"comment":"The heading of Section VI reads \"STRATERGY\" instead of \"Strategy.\"","section":"Section VI heading"},{"comment":"Equation (5) is not clearly defined: dividing PageRank(pi)/OutDegree(pi) by EdgeWeight(V1, pi) without normalizing by the total edge weight is dimensionally odd and is not described in the text; the sentence before the equation mentions dividing by a summation that does not appear in the formula.","section":"Eq. (5)"},{"comment":"The density window length (150 samples), PageRank damping factor d, and graph window size Δt are introduced without justification or sensitivity analysis; these are free parameters of the method.","section":"§VI-C, §VI-D"},{"comment":"Several references lack complete bibliographic information, and reference [60] contains a \"utm_source=chatgpt.com\" tracking parameter in its URL, which is inappropriate for an archival reference list.","section":"References"},{"comment":"The paper does not include a data/code availability statement, which would be useful for verifying the reported results.","section":"General"}],"recommendation":"reject","confidential_remarks":"I agree with the reader's assessment. The central claim cannot be fixed by local edits: either the Transfer ID dependency must be removed and all experiments re-run, or the protocol-independence claim must be dropped and the contribution re-scoped. Either path goes beyond a normal revision. In addition, the reference list contains a tracked marketing URL ([60]) and several incomplete citations; if the authors resubmit a substantially revised version, those issues should also be cleaned up."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe short version: this is a useful, incremental empirical study with a specific graph-construction trick, but the paper's central claim—that the IDS works 'without using any decoded features based on the UAVCAN protocol'—is contradicted by its own method, and the reported evaluation is too thin to support the comparison with the LSTM baseline.\n\nWhat's actually new: the authors build UAVCAN message graphs using timestamp-weighted edges between consecutive CAN IDs, add a self-loop between identical IDs only when the transfer ID changes (to separate multi-frame messages), and use PageRank plus density as node features. They then compare GCNN, GAT, GraphSAGE, and a graph transformer on the public HCRL UAVCAN dataset. That combination hasn't been tested on this dataset before, and the inductive-vs-transductive comparison is a legitimate extension. The paper credits prior graph-based CAN IDS work fairly.\n\nThe soft spots are real. The protocol-independence claim is load-bearing and fails on a close read: Section VI-A says one feature, transfer ID, is used for graph construction, and that field is extracted from the UAVCAN payload tail byte after decoding. Section VI-B's self-loop constraint depends on it. The HCRL dataset ships already-decoded messages, so the authors didn't have to decode anything themselves, but a deployed protocol-independent IDS would not have a transfer ID column. Remove it and the graph changes for multi-frame messages. That contradiction is in the abstract, the introduction, and the conclusion, so it's not a side remark.\n\nThe performance side is also shaky. In Scenarios 2 and 4 (slower flooding and fuzzy attacks), all four graph models report precision, recall, and F1 around 0.5 while accuracy sits above 0.97—classic near-random detection on the attack class under imbalance. There are no error bars, no repeated runs, no explicit train/test splits, and no hyperparameters. The 'LSTM baseline' is taken from an external paper, not retrained under matched conditions, so the 'all graph models outperform LSTM' claim isn't a fair comparison. No code or data release either.\n\nBottom line: the graph-construction idea is worth knowing about, but the paper as written overclaims. If the authors dropped the protocol-independence claim, retrained the LSTM, and reported class-wise metrics with variance, the core would be worth publishing. As is, I wouldn't accept, but I'd send it to a serious referee to force those changes. I'd probably bring it to a reading group as a cautionary example of a good idea wrapped in an unsupported general claim.","headline":"Useful graph construction for UAVCAN intrusion detection, but the protocol-independence claim fails on the method's own use of transfer ID, and the metrics are too thin to support the comparison.","tokens_in":18724,"tokens_out":3913,"would_cite":false,"duration_ms":37630,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Graph-based models detect drone CAN-bus injection attacks without decoding the payload.","keywords":["Controller Area Network","intrusion detection system","graph neural networks","UAVCAN protocol","unmanned aerial vehicle security","message injection attacks","protocol-independent detection","PageRank"],"falsifier":"Run the identical graph construction on raw CAN frames with the transfer ID removed (or on a CAN-based system with no equivalent field), using only CAN ID, timestamps, and the self-loop rule replaced by a timing threshold, and compare detection on multi-frame replay attacks against the LSTM baseline; if accuracy drops to near-baseline or below, the protocol-independence claim is falsified. A complementary check: craft an injection attack whose fake messages reuse legitimate CAN IDs and transfer-ID plus-minus patterns so the density and PageRank statistics stay inside the normal range, and test whether the detector still flags it.","tokens_in":17707,"feed_emoji":"🛸","tokens_out":6459,"duration_ms":58810,"temperature":0.7,"pith_summary":"The paper sets out to build a lightweight, protocol-independent intrusion detection system for the Controller Area Network (CAN) bus inside unmanned aerial vehicles. It converts sequences of UAVCAN messages into graphs—nodes are distinct CAN identifiers, edges follow the order messages appear, and edge weights record timing gaps—then feeds node features (PageRank and per-CAN-ID density) into four graph-based machine learning models. The central claim is that these graph models catch flooding, fuzzy, replay, and mixed injection attacks better than a baseline LSTM that does decode the UAVCAN payload, with accuracy gains of 1–29 percent, and that the best results come from inductive models like GraphSAGE and graph transformers. If true, drone makers could deploy intrusion detection without reverse-engineering or disclosing their CAN message formats, and the same pipeline could transfer to other CAN-based vehicles.","feed_headline":"Graph models catch drone bus attacks without reading payloads","feed_subtitle":"On drone CAN logs, GraphSAGE and graph transformers beat a payload-decoding LSTM by up to 29 percent.","key_machinery":"The load-bearing construction is the graph stream built from raw CAN traffic: within each fixed window of 100 samples, each unique CAN ID becomes a vertex, a directed edge connects consecutive CAN IDs, consecutive identical CAN IDs get a self-loop only when the UAVCAN transfer ID changes (marking a new multi-frame message), and the edge weight is the summed timestamp gap between the two endpoints. On top of this graph, the pipeline computes a modified PageRank that incorporates edge weights and a density feature counting each CAN ID's occurrences over the current window plus the previous 150 samples. These per-vertex features are what the graph neural networks classify, so the self-loop rule is what lets a 'protocol-independent' detector still separate repeated message frames from new messages.","core_discovery":"The paper claims that converting CAN bus traffic into a temporal graph—vertices are CAN IDs, edges follow message order, edge weights accumulate timestamp gaps, and self-loops mark new messages when the UAVCAN transfer ID changes—lets graph neural networks detect flooding, fuzzy, replay, and mixed injection attacks on drone CAN buses more accurately than an LSTM trained on decoded UAVCAN payload data. Across ten attack scenarios, the inductive graph models (GraphSAGE and a graph transformer) stayed above about 90 percent accuracy in most scenarios, with gains of roughly 1 to 29 percentage points over the LSTM baseline, and they do this without decoding the payload contents. The authors take this as evidence that the detection signal lives in the structure and timing of CAN IDs rather than in the protocol payload, making the approach portable to proprietary or undocumented CAN protocols.","pith_inferences":["The protocol-independence claim is narrower than it sounds: the self-loop rule reads the transfer ID from a decoded UAVCAN tail byte, so the method is payload-independent but not decoding-free. A truly closed protocol would need a proxy for transfer ID, such as timing gaps between identical IDs, and that proxy's reliability is untested.","Because the features are statistical summaries of ID order and frequency, an attacker who paces injected frames to mimic normal CAN ID distributions and transfer-ID patterns could plausibly evade the detector; this evasion scenario is not explored in the paper.","A natural testable extension is to run the same graph construction on automotive CAN data (11-bit IDs, no UAVCAN transfer ID) to see whether the self-loop rule can be replaced by inter-frame timing; success there would support the generalization argument.","The density window (150 prior samples) and graph window (100 samples) are fixed rather than tuned per attack; varying these windows might improve detection of slow replay attacks, where the current numbers show lower precision."],"forward_implications":["A CAN-based drone IDS can be built without access to payload contents, so it applies to proprietary or undocumented protocols.","Inductive graph models (GraphSAGE, graph transformers) are the better choice for dynamic CAN traffic because they generalize to new graphs, while transductive GCNNs lag on some scenarios.","Replacing a payload-decoding LSTM with graph features buys 1 to 29 percentage points of accuracy across all ten tested attack scenarios.","The feature set—CAN ID ordering, density, and timing-weighted PageRank—is cheap enough to run on microcontrollers rather than GPU-equipped boards."],"supporting_citations":[{"why":"Supplies the UAVCAN drone CAN bus dataset and the ten attack scenarios (flooding, fuzzy, replay, mixed) used in all experiments.","marker":"[27]"},{"why":"Provides the PageRank algorithm that the paper modifies with edge weights to produce node importance features.","marker":"[58]"},{"why":"Defines the decoded-payload LSTM baseline whose accuracy the graph models are compared against.","marker":"[59]"},{"why":"Introduces a graph convolutional network for CAN bus intrusion detection, the transductive baseline this paper extends with inductive models.","marker":"[56]"},{"why":"Anchors the density-of-reception-cycle-periods feature that the paper adapts as its per-CAN-ID density feature.","marker":"[43]"},{"why":"Supports the graph-based IDS approach for CAN buses, used as motivation for combining graph analysis with learned detection.","marker":"[6]"}],"fun_headline_variants":["Graph models beat LSTM at drone CAN bus attack detection","Drone CAN attacks caught by graph models, not payload decoding","Graph neural nets spot drone CAN intrusions without reading payloads","For drone CAN security, graph models outperform LSTM without payload decode","Graph structure alone catches drone CAN bus attacks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim of protocol independence rests on using the UAVCAN transfer ID—a field taken from decoded protocol messages—to decide when two consecutive identical CAN IDs are separate messages; if that field is unavailable, the graph loses multi-frame message boundaries and the central claim collapses.","fun_headline_variants_meta":{"raw":{"variants":["Graph models beat LSTM at drone CAN bus attack detection","Drone CAN attacks caught by graph models, not payload decoding","Graph neural nets spot drone CAN intrusions without reading payloads","For drone CAN security, graph models outperform LSTM without payload decode","Graph structure alone catches drone CAN bus attacks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000704,"raw_usage":{"total_tokens":3214,"prompt_tokens":1025,"completion_tokens":2189,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":641,"completion_tokens_details":{"reasoning_tokens":2107}},"tokens_in":641,"tokens_out":2189,"duration_ms":14673,"temperature":1.0,"reasoning_tokens":2107,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T23:20:25.650475+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the identical graph construction on raw CAN frames with the transfer ID removed (or on a CAN-based system with no equivalent field), using only CAN ID, timestamps, and the self-loop rule replaced by a timing threshold, and compare detection on multi-frame replay attacks against the LSTM baseline; if accuracy drops to near-baseline or below, the protocol-independence claim is falsified. A complementary check: craft an injection attack whose fake messages reuse legitimate CAN IDs and transfer-ID plus-minus patterns so the density and PageRank statistics stay inside the normal range, and test whether the detector still flags it.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Anchors the density-of-reception-cycle-periods feature that the paper adapts as its per-CAN-ID density feature."},{"cited_title":"UAVCAN Dataset Description","cited_arxiv_id":"2212.09268","evidence_quote":"Supplies the UAVCAN drone CAN bus dataset and the ten attack scenarios (flooding, fuzzy, replay, mixed) used in all experiments."},{"cited_title":"Brin and L","cited_arxiv_id":null,"evidence_quote":"Provides the PageRank algorithm that the paper modifies with edge weights to produce node importance features."},{"cited_title":"Tlili, S","cited_arxiv_id":null,"evidence_quote":"Defines the decoded-payload LSTM baseline whose accuracy the graph models are compared against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supports the graph-based IDS approach for CAN buses, used as motivation for combining graph analysis with learned detection."}],"review_version":1}