{"id":"83f43176-4ef9-42f8-8024-1d47e171b232","arxiv_id":"2607.18108","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"LLMs with automotive-CTI retrieval reconstruct high-level tactical attack paths for unseen vehicles but fail at implementation-level steps — a measured automation boundary for TARA support.","lead":"GARAGE is a retrieval-augmented pipeline that turns 12,786 automotive CVEs and 140 incident reports into a threat-intelligence knowledge base and generates vehicle attack graphs using eight LLMs. Tested by deleting the target attack from the knowledge base across four famous car hacks (Jeep, Mercedes, Nissan, BMW), it shows LLMs can sketch tactical-level attack paths but fail at implementation-specific steps.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"LOO protocol leaves target-associated CVEs in the KB, so 'unseen architecture' transfer is not established; a contamination probe is needed.","rationale":"The reader's central concern — that 'unseen' is not established because the LLM's parametric knowledge of four famous attacks is not erased by the LOO protocol — is correct and load-bearing. My stress-test sharpens this into a more specific, internally checkable flaw: the LOO protocol removes incident reports but not the CVE entries that encode the same attacks. The paper's own data (Table 3, App. A.1) makes the leak plausible and consequential. NO_RAG performing as well as Hybrid shows parametric recall is sufficient; if explicit CVE knowledge also remains in the KB, the 'entirely unseen' claim rests on no controlled removal at all. I therefore agree with the reader's bottom line — CONDITIONAL — but my mechanism is narrower and more directly falsifiable. The per-step boundary characterization (tactical high coverage vs. implementation low coverage, Jeep persistence 0/10) is well-supported and should stand; the concern is specifically about attributing transfer to GARAGE's RAG/KB pipeline and to 'unseen' generalization. A contamination probe would settle this without discarding the paper's useful empirical boundary. Because the reader already recommended CONDITIONAL, my verdict remains UNCHANGED.","tokens_in":24207,"tokens_out":4042,"duration_ms":40827,"concrete_test":"Inspect each of the 16 LOO KB instances for the presence of the four incidents' associated CVE IDs and target-specific artifacts listed in Table 3 (e.g., CVE-2017-7932, libevo_stack.so, /fs/mmc0, cmcioc.bin). Run the exact top-10 retrieval pipeline from §2.4.1 with the ATM-guided queries; if any target-associated CVE or artifact is retrieved or present in the KG after filtering, recompute per-case PF with those entries removed. If PF drops substantially for the affected cases, the 'unseen architecture transfer' claim is an artifact of incomplete LOO rather than cross-vehicle knowledge transfer.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.1's Leave-One-Out procedure removes the target incident report and filters queries/retrieved results for target-specific keywords (jeep, uconnect, bmw, connecteddrive), but the KB also contains 12,786 CVE descriptions built in §2.2.1. The four benchmark attacks rely on publicly documented CVEs — for example, Nissan's ground truth in Table 3 lists CVE-2017-7932 and libevo_stack.so as key entities. Nothing in the LOO protocol removes CVE entries associated with the four attacks; keyword filtering cannot catch a CVE ID or a CVE text that omits the vehicle name. The CVE corpus is thus a second, unremoved channel through which exact target-attack knowledge can enter retrieval. This is not merely parametric-memory contamination: it is a concrete gap in the stated LOO methodology. The paper's own ablation (App. A.1) shows NO_RAG ≈ Hybrid (PF 47.8 vs 46.6), so the LLM's parametric knowledge alone reproduces the measured performance; if the vector store and KG still contain the relevant CVEs, RQ1's claim of transfer to 'entirely unseen vehicle architectures' is doubly compromised. The boundary finding about tactical vs. implementation-level steps may survive, but the headline generalization is not cleanly supported until this leak is ruled out.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents GARAGE, a hybrid RAG pipeline that converts 12,786 automotive CVEs and 140 incident reports into a STIX/Auto-ISAC-aligned knowledge base, then generates vehicle-level attack graphs for user-supplied vehicle architectures. The main empirical claim is that, under Leave-One-Out (LOO) evaluation with entity anonymization, GARAGE accurately transfers security knowledge to entirely unseen vehicle architectures; the more nuanced contribution is an automation boundary: tactical-level threat scenarios can be generated automatically, while implementation-detail steps require human experts. The evaluation uses a dual PF/KR metric judged by Claude 4.5 Sonnet, 8 target LLMs, 4 attack cases, and 10 iterations per condition. The paper also provides a cost-performance analysis and an ablation isolating retrieval modality.","tokens_in":24497,"tokens_out":3529,"duration_ms":44726,"significance":"If the 'unseen architecture' claim were cleanly established, this would be a useful contribution to automotive TARA: a public domain KB, a dual-metric evaluation scheme, and fine-grained step-level evidence for where LLM automation stops. The paper is unusually candid in its limitations, and several empirical patterns (e.g., Jeep persistence 0/10, Execution tactic 15% coverage, KB Match Rate ranking cases) are internally consistent and support the automation-boundary claim regardless of the transfer claim. However, the headline transfer claim is currently not supported because the LOO protocol leaves a concrete retrieval channel open, and the authors' own ablation shows that parametric knowledge alone matches the RAG pipeline. The significance of the work thus depends on whether the contamination leak can be ruled out and the claims accordingly recalibrated.","major_comments":[{"comment":"The LOO protocol is incomplete for the 'entirely unseen architecture' claim. §3.1 removes the target incident report and filters queries/retrieved results for make/model keywords, but §2.2.1 populates the KB with 12,786 CVE descriptions. The four benchmark attacks rely on publicly documented CVEs; e.g., Table 3 lists CVE-2017-7932 for Nissan. Nothing removes these CVE records from the vector store or KG, and keyword filtering cannot exclude a CVE ID or a CVE description that omits the vehicle name. This is a second, unremoved channel through which exact target-attack knowledge enters retrieval. The ablation in Table 11 (NO_RAG PF 47.8 vs Hybrid 46.6) makes this concrete: parametric memory alone reproduces the measured performance, so the observed scores cannot be attributed to KB-grounded cross-vehicle transfer. The authors should either (a) re-run the experiments with all target-case CV","section":"§3.1, §2.2.1, App. A.1"},{"comment":"The LLM-as-a-judge setup has a residual circularity that is only partially mitigated. Claude 4.5 Sonnet is both a top-performing target model and the primary judge; the cross-judge validation in Table 10 covers only two of four cases (C1, C4; N=160), not the full 320-instance evaluation. The four Extraction & Inference models are also evaluated on KB instances they themselves constructed, so KB quality is not independent of the model being scored. The paper should provide cross-judge agreement for all four cases or a human-annotated subset, and explicitly state whether the self-built-KB design is intentional and what bias it could introduce.","section":"§3.3.2, L2, Table 10"},{"comment":"The paper repeatedly claims that the KB-grounded pathway provides formal evidence traceability, but this is unquantified and partly contradicted by the authors' own data. §5.1 says 'Quantifying the grounding rate of each generated step remains future work,' and L3 reports 76.9% of attack graphs contained at least one hallucinated element, with a mean Knowledge Precision of 9.81/20 — approximately half of the generated knowledge elements lack direct traceability to retrieved context. Until a per-step grounding rate is reported and hallucinated elements are excluded, the manuscript should not describe the output as 'traceable evidence.' This does not undermine the automation-boundary finding, but it weakens the regulatory-compliance framing.","section":"§5.1, L3"}],"minor_comments":[{"comment":"The text states Affect Vehicle Function coverage is 98.8%, while Figure 3's y-axis label shows 0.99 (99%). Reconcile the two values.","section":"§4.1.2, Fig. 3"},{"comment":"The header 'V ALID Rate' has an extra space; also unify whether the table reports per-step VALID rates as percentages or decimals.","section":"Table 4"},{"comment":"The caption does not say which iteration or model produced the bottom graph, nor whether it is the best-scoring example. For consistency with Figures 7–10, state this explicitly.","section":"Fig. 6"},{"comment":"Section 2.3 says all KB construction stages are performed by the same four LLMs, but §2.3.6 uses a separately fine-tuned GPT-4.1 model for ATM technique classification. Clarify that the technique classifier is an additional component, not one of the four extraction models.","section":"§2.3, §2.3.6"},{"comment":"The sentence 'Since each model produces a unique knowledge representation, we created 16 hybrid KB instances (4 Extraction Models × 4 scenarios)' is confusing: scenario-specific KBs are not described earlier. Explain how the 16 instances relate to the general KB construction in §2.2–2.3.","section":"§3.2.1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript's central automation-boundary result appears defensible and is supported by step-level evidence. The obstruction is the RQ1 'unseen architecture' claim, which is compromised by the unremoved CVE channel and by the NO_RAG ablation. This is fixable within the manuscript's scope, but it requires either a rerun with complete target-case CVE removal and contamination probes, or a substantially weakened claim. I would not recommend rejection if the authors perform that revision; the paper is otherwise systematic and its limitations sections are unusually honest."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"GARAGE is worth a serious look, but the paper's headline is doing more work than its evidence supports. The durable result is the automation boundary: LLMs can reconstruct tactical-level attack paths across vehicle cases, but fail on implementation-specific steps. That claim is supported by per-step data (Jeep persistence 0/10 in all eight models, Execution tactic 15% coverage) and the KB Match Rate -> PF pattern. The released KB is a real artifact, and the dual-metric evaluation is a reasonable way to separate path viability from knowledge depth.\n\nThe problem is the 'entirely unseen architecture' framing. The LOO protocol removes the target incident from the stores and filters queries/results for keywords like 'jeep' and 'uconnect', but the KB still holds 12,786 CVE descriptions built from the same public sources. The Nissan ground truth explicitly includes CVE-2017-7932; nothing in the protocol removes that CVE from the vector store or KG. Keyword filtering can't catch a CVE ID or a CVE description that doesn't mention the vehicle name. So the target attack is not fully absent, and the measured transfer is confounded.\n\nThe paper's own ablation makes this worse: NO_RAG scores essentially the same as the full Hybrid pipeline (47.8 vs 46.6 PF). The authors interpret this as evidence that knowledge granularity dominates retrieval modality, and they're honest that the RAG ground path matters for traceability, not score. But it also means the framework's specific contribution—the KB plus RAG—is not shown to improve attack graph quality. The performance is coming from the LLM's parametric knowledge, which is exactly what LOO was supposed to control. This is not a fatal flaw for the boundary finding, but it does mean the 'framework can accurately transfer' claim is unsupported as stated.\n\nThere is also the LLM-as-judge circularity: Claude 4.5 is both a top target and the primary judge. Cross-judge agreement covers only two of four cases and is moderate (rho ~0.74-0.84). And the hallucination numbers are sobering: 76.9% of graphs contain at least one invented element, KR precision sits at 9.81/20. Those are acknowledged in L3, but they should temper the abstract's 'accurate'.\n\nWhere the paper earns credit: it ships a substantial KB, the per-step analysis is reproducible-looking, the arithmetic is internally consistent, and the limitations section is unusually candid. The automation-boundary result—tactical patterns transfer, implementation details don't—survives the contamination concern because it rests on step-level behavior, not on LOO purity.\n\nWho benefits: TARA practitioners and researchers building LLM-based threat enumeration. They should read the boundary result and take the framework claim with a pinch of salt. I'd like to see a contamination probe (synthetic incidents or permuted KB) before trusting the 'unseen' claim. Send to review, yes—but with a request for claim revision and that probe.","headline":"A well-supported automation-boundary result buried under a headline claim that the LOO protocol doesn't actually support.","tokens_in":25154,"tokens_out":2452,"would_cite":true,"duration_ms":391608,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"GARAGE claims that LLMs can reconstruct high-level attack paths for vehicle architectures never seen in their knowledge base, while low-level implementation details still require human experts.","keywords":["attack graph generation","large language models","retrieval-augmented generation","automotive security","threat analysis and risk assessment","leave-one-out evaluation","knowledge granularity","human-in-the-loop"],"falsifier":"A concrete falsifier: repeat the exact leave-one-out protocol on a fifth attack that postdates the training corpora of all eight models (or on a synthetic vehicle with brand-new components and a synthetic exploit with no public footprint). If practical-feasibility scores collapse to near zero, the reported cross-architecture transfer was mostly parametric recall, not architectural reasoning; if high scores persist, the automation-boundary claim is robust.","tokens_in":23926,"feed_emoji":"🚗","tokens_out":8352,"duration_ms":91374,"temperature":0.7,"pith_summary":"The paper introduces GARAGE, a retrieval-augmented pipeline that turns public automotive vulnerability and incident data into a structured knowledge base, then prompts large language models to generate multi-step attack graphs for a target car. Its central claim is an automation boundary: LLMs can produce high-level tactical threat scenarios for vehicle architectures that were deliberately excluded from the knowledge base, but low-level implementation details—specific firmware, filesystem paths, and protocols unique to one vehicle—require human experts. The claim rests on 320 leave-one-out experiments across four well-documented real-world hacks and eight models, with proprietary models reconstructing valid attack paths at a mean practical-feasibility score near 59 out of 100. A sympathetic reader would care because, if correct, automotive threat analysis and risk assessment (TARA) can be front-loaded with machine-generated candidate paths while keeping expert review for feasibility.","feed_headline":"LLMs sketch attack paths for unseen car hacks—but only the tactics","feed_subtitle":"A 320-run leave-one-out study maps where automated threat modeling stops and human experts must take over.","key_machinery":"The load-bearing mechanism is the leave-one-out-plus-anonymization protocol: for each of four real-world attacks, all documents about the target are removed from both the graph and vector stores, and target-specific keywords are filtered from queries and retrieved results, so any successful reconstruction must come from cross-vehicle generalization or residual parametric memory. Around that protocol sits the hybrid RAG engine—structured graph traversal for component-to-vulnerability connectivity plus vector retrieval for semantic evidence—and a dual-metric judge that separates path viability from knowledge depth. This design is what lets the paper attribute performance differences to knowled","core_discovery":"GARAGE is an end-to-end framework that converts 12,786 CVEs and 140 incident reports into a structured automotive-security knowledge base—a graph of roughly 8,800 entities and 9,200 relations alongside a vector store of rewritten evidence—and then uses hybrid retrieval, combining graph traversal with semantic search, to condition a large language model to write multi-step attack graphs for a given vehicle specification. The core empirical claim is an automation boundary: with the target incident removed from both stores and target-specific names filtered from queries, proprietary models reconstructed high-level tactical attack paths (mean practical-feasibility score near 59/100), while every","pith_inferences":["If the tactical/implementation boundary is a general property of security LLMs, the same split should appear in other regulated domains (medical devices, industrial control systems); a cross-domain replication of the leave-one-out protocol would test that.","Because ablation shows retrieval modality barely moves scores, a leaner alternative could skip the curated graph and simply require the LLM to cite its sources, then measure whether regulators accept citation-level grounding as compliance evidence.","Quantifying per-step grounding—which the paper leaves as future work—would create a practical trust cutoff: high-grounding steps skip review, low-grounding steps go to human experts.","The strong cost-performance of small open-weight models on tactical enumeration suggests air-gapped automotive teams could run initial threat scans locally and reserve premium APIs for high-stakes confirmations."],"forward_implications":["TARA workflows can open with machine-generated tactical attack candidates, and the paper shows high coverage specifically at the entry-point and safety-impact stages, where analysts may otherwise miss scenarios.","Implementation-heavy stages—execution, command-and-control, persistence—remain human tasks, so the realistic deployment is a human-in-the-loop assistant rather than a fully autonomous analyzer.","When public knowledge about an architecture is rich, even small open-weight models approach proprietary performance; when coverage is sparse, model choice becomes decisive.","The knowledge base contributes less to raw accuracy than to evidence traceability: a no-retrieval baseline scores comparably, so GARAGE's case for deployment rests on auditability for regulatory compliance.","Cost-per-analysis results give concrete tier guidance: cheap models for large-scale enumeration, premium models for accuracy-critical compliance work."],"fun_headline_variants":["Car-threat LLM: tactic-level attack paths yes, technique-level no","GARAGE: 12,786 CVEs become attack graphs—but only the blueprint","AI attack graphs for unseen cars: tactical wins, technical limits","Automotive threat graphs: high-level tactics, not specific exploits"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that removing the target incident from the knowledge stores and filtering target-specific keywords truly makes the attack 'unseen' to the LLM—since the four test hacks are among the most widely documented automotive exploits, their details almost certainly persist in the models' pretrained weights, so the measured 'transfer' may be partly remembered rather than inferred.","fun_headline_variants_meta":{"raw":{"variants":["Car-threat LLM: tactic-level attack paths yes, technique-level no","GARAGE: 12,786 CVEs become attack graphs—but only the blueprint","AI attack graphs for unseen cars: tactical wins, technical limits","Automotive threat graphs: high-level tactics, not specific exploits"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000223,"raw_usage":{"total_tokens":1261,"prompt_tokens":680,"completion_tokens":581,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":424,"completion_tokens_details":{"reasoning_tokens":501}},"tokens_in":424,"tokens_out":581,"duration_ms":6886,"temperature":1.0,"reasoning_tokens":501,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T15:59:42.217094+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete falsifier: repeat the exact leave-one-out protocol on a fifth attack that postdates the training corpora of all eight models (or on a synthetic vehicle with brand-new components and a synthetic exploit with no public footprint). If practical-feasibility scores collapse to near zero, the reported cross-architecture transfer was mostly parametric recall, not architectural reasoning; if high scores persist, the automation-boundary claim is robust.","supporting_citations":[],"review_version":1}