{"id":"12b7c9e4-648f-446d-9912-40e475e3ebdd","arxiv_id":"2607.03135","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"low","formal_verification":"none","parameter_count":0,"one_line_summary":"Hardware traces abstracted to message exchanges, deterministically differenced against design sequence diagrams, detect architectural drift (confirmed/missing/additional/inverted) in ISO 26262 firmware, with constrained LLM reports for review.","lead":"This paper gives a practical way to catch when safety-critical firmware no longer matches its design diagrams by comparing hardware traces to sequence diagrams. It matters because long-lived automotive firmware drifts silently and ISO 26262 audits need evidence of that gap.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"No significant objection identified beyond the paper's own scope limits.","rationale":"The reader's weakest_assumption correctly isolates the linear-sequence premise (Section III) that makes inverted classification reliable and excludes UML combined fragments. That premise is load-bearing for the inverted category and for applicability claims, yet the paper states it openly, evaluates only linear diagrams, and still obtains strong metrics on the other three categories that dominate the practical claim. No deeper internal flaw (e.g., circular ground-truth construction, LLM leakage into delta computation, or unacknowledged non-determinism) is evident. Therefore the CONDITIONAL verdict with high confidence remains appropriate; no adjustment is warranted.","tokens_in":17017,"tokens_out":448,"duration_ms":4905,"concrete_test":"Independently re-implement the Differ on the 26 PlantUML pairs (or a public synthetic suite of linear sequences with known injected missing/additional/inverted messages) and recompute precision/recall/F1; if confirmed/missing/additional F1 remain ≥ 85 % and inverted ≥ 55 %, the load-bearing detection claim is corroborated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim holds within the stated premises. The methodology (Phases 1–2) deterministically reduces HIT traces and design-time sequence diagrams to linear message triples (source, target, label) and classifies them as confirmed/missing/additional/inverted; the LLM is confined to reporting (Phase 3, Box 1). RQ1 metrics on 26 industrial cases (F1 94.6 % confirmed, 88.2 % missing, 86.4 % additional; 60 % inverted) and substantial expert agreement on root-cause faithfulness (κ = 0.692) support the practical-utility claim. The sequential/linear-sequence restriction (Section III) is the weakest premise, but it is explicitly scoped, justified by high-integrity deterministic scheduling, and already flagged by the reader; it does not create an internal inconsistency or invalidate the reported results inside that scope. Proprietary data and single-organization survey limit generalizability, yet the paper does not claim broader applicability.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper proposes a three-phase methodology for detecting interaction-level architectural drift in ISO 26262-compliant firmware. Hardware-assisted instruction traces are abstracted into linear message exchanges among components (Phase 1), compared deterministically against design-time UML sequence diagrams to produce a labeled delta of confirmed, missing, additional, or inverted interactions (Phase 2), and summarized by a constrained LLM that is forbidden from altering the delta (Phase 3). Evaluation on 26 industrial feature-linked test cases reports high precision/recall/F1 for confirmed (F1 94.6%), missing (88.2%), and additional (86.4%) elements, lower performance for inverted (60%), substantial expert agreement on LLM root-cause faithfulness (Fleiss’ κ = 0.692), and positive TAM-style practitioner ratings from 23 engineers on usefulness and compliance support.","tokens_in":17274,"tokens_out":1084,"duration_ms":29375,"significance":"If the results hold, the work offers a practical, industrially grounded pipeline that couples non-intrusive HIT traces with deterministic architectural differencing and carefully scoped LLM reporting—an attractive pattern for safety-critical settings where reproducibility and auditability matter. Strengths include an explicit needs elicitation at Micron, a clear separation of the Differ engine from the LLM (Box 1), agreement-based validation against majority-voted expert ground truth, and a practitioner survey that addresses perceived usefulness for ISO 26262 documentation. The contribution is incremental rather than foundational, but it usefully bridges static architecture recovery and runtime evidence for long-lived embedded firmware within a well-stated sequential-execution scope.","major_comments":[{"comment":"Section V-D-1 / RQ1 vs. Abstract: RQ1 is framed as alignment of the “generated architectural-drift delta” with an expert reference, yet the procedure constructs ground-truth sequence diagrams by source-code analysis and then classifies elements of the methodology’s reconstructed runtime diagram against that ground truth. This measures Phase-1 reconstruction fidelity more directly than design-time-vs-runtime drift detection. Please clarify (i) whether the 26 cases contained known design–runtime discrepancies, (ii) how expert-curated delta references (if any) were built, and (iii) how the reported F1 numbers should be read as evidence for drift detection rather than only for trace abstraction accuracy.","section":null},{"comment":"Section V-E (RQ1 inverted class): F1 = 60% for inverted interactions is substantially weaker than the other categories, and absolute counts are withheld for confidentiality. Without base rates or a confusion matrix it is hard to judge whether inverted detection is systematically unreliable or merely sparse. Because ordering is central to the paper’s definition of inverted drift (Section III, Section IV-D), please either report instance counts / prevalence (even as ranges) or strengthen the discussion of why ordering discrepancies remain hard and how this limits the claim of “strong agreement” overall.","section":null},{"comment":"Section III (linear-sequence premise) and evaluation scope: The methodology and the 26 test cases are restricted to already-linear design-time diagrams (no alt/opt/loop). This is stated, but the abstract and conclusions still present the approach as supporting architectural drift detection in evolving ISO 26262 firmware more generally. Please make the linear-scenario restriction equally prominent in the abstract/claims, or provide at least a qualitative discussion of how many industrial scenarios fall outside this premise and what that implies for adoption.","section":null}],"minor_comments":[{"comment":"Throughout: occasional spacing artifacts in technical terms (“DW ARF”, “M odels@run.time”) and reference encoding issues (e.g., Altınıs ¸ık) should be cleaned for camera-ready.","section":null},{"comment":"Box 1 prompt: “use less word possible” and similar phrasing should be polished; the prompt is part of the method description and will be read carefully.","section":null},{"comment":"Figure 9: Likert distributions are useful; adding mean/median or a short table of response counts would improve readability without violating confidentiality.","section":null},{"comment":"Section V-C: state more explicitly that selected diagrams contained no UML combined fragments so readers do not infer that the firmware itself lacks conditionals.","section":null},{"comment":"Related work: the distinction from Models@run.time and from LLM-based architecture recovery (Hatahet et al.) is clear; a one-sentence comparison to classic architecture-reflexion / drift literature (e.g., Perry & Wolf, Tekinerdogan) would help position the interaction-level definition.","section":null}],"recommendation":"minor_revision","confidential_remarks":"Solid industrial SE paper with a sensible deterministic-core + constrained-LLM design. The evaluation-framing ambiguity (reconstruction vs. true design–runtime delta) is the main fixable issue; once clarified, minor revision should suffice. Single-organization proprietary setting is typical for this venue class and is adequately caveated. No integrity or novelty concerns."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"This is a practical engineering paper, not a theory one. What is new is the packaged pipeline: hardware-assisted instruction traces abstracted to inter-component message triples, a deterministic four-way differ (confirmed / missing / additional / inverted) against design-time PlantUML sequences, and an LLM confined to post-hoc reporting under a tight prompt. That separation is the right design choice for ISO 26262 work, and the authors are explicit about it.\n\nThey do the evaluation carefully for an industrial setting. Twenty-six feature-linked cases, majority-voted expert ground truth, high F1 on confirmed/missing/additional (roughly 86–95 %), substantial kappa (0.692) on root-cause faithfulness, and a 23-person TAM-style survey that is mostly positive. The threats section and lessons-learned are honest about proprietary data, single-organization scope, and the weaker inverted class (60 %). Citations cover static recovery, runtime verification, Models@run.time, and prior architecture-recovery work without padding.\n\nThe soft spots are real but already scoped. Everything assumes linear, deterministic sequences—no alt/opt/loop fragments, only first-level intra-module calls—so ordering inversions are only reliable under that premise. That matches high-integrity scheduling, yet it limits how far the method travels into concurrent or heavily conditional firmware. Survey generalizability is modest; inverted detection is the clear weak metric. None of this breaks the central claim inside the stated bounds.\n\nWho it is for: people who maintain long-lived automotive/safety firmware and need inspectable design-vs-runtime evidence for reviews and audits. A serious referee should see it; the contribution is concrete, the evaluation is proportionate, and the limitations are not hidden. I would engage with it and expect revision mainly around broader fragment handling and stronger external validation, not a rewrite of the core idea.","headline":"Solid industrial pipeline that turns HIT traces into a four-class architectural delta and keeps the LLM strictly out of the classification step; results hold inside a clearly scoped sequential model.","tokens_in":17826,"tokens_out":480,"would_cite":true,"duration_ms":6278,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Hardware traces plus deterministic message differencing can flag architectural drift in safety-critical firmware, with an LLM only writing the review report.","keywords":["Architectural Drift","Architecture Evolution","Software Architecture Recovery","ISO 26262","Embedded Firmware","Execution Trace Analysis","Sequence Diagrams","Runtime Verification"],"falsifier":"On a held-out set of industrial feature-linked test cases whose design diagrams are linear, measure whether the automatically labeled deltas still reach high F1 against independent expert ground truth for confirmed, missing, and additional classes, and whether practitioners still rate the reports useful when the LLM step is removed or replaced by a non-LLM template.","tokens_in":17960,"feed_emoji":"🔍","tokens_out":880,"duration_ms":9593,"temperature":0.7,"pith_summary":"Long-lived safety-critical firmware often drifts away from its design documents: components start exchanging different messages, skip expected ones, or reverse order, yet the paper trail lags. This paper claims that the drift can be made visible by collecting non-intrusive hardware-assisted execution traces, abstracting them into the same vocabulary of component lifelines and messages used in design-time sequence diagrams, and then running a deterministic differencing step that labels every interaction confirmed, missing, additional, or inverted. A constrained language model is allowed only to turn the already-computed delta and related commit stories into a short human-readable report; it never decides what the delta is. On 26 industrial feature-linked test cases the generated deltas agreed strongly with expert-curated references, and practitioners rated the reports useful for interpretation, effort reduction, and safety-oriented documentation. If the claim holds, review teams gain an inspectable, auditable signal of design-runtime mismatch without treating the language model as primary safety evidence.","feed_headline":"Traces catch firmware design drift before safety reviews miss it","feed_subtitle":"Deterministic message diffs flag missing, extra, or inverted calls; LLM only writes the report","key_machinery":"The Delta Trace: an ordered collection of source-target-message records that the Differ matches across design-time and runtime PlantUML sequences, labeling each exchange confirmed (present and order-consistent), missing, additional, or inverted.","core_discovery":"The authors establish that interaction-level architectural drift in ISO 26262-compliant firmware can be detected by elevating hardware-assisted instruction traces to linear message-exchange sequences, comparing them deterministically against design-time sequence diagrams, and classifying each discrepancy as confirmed, missing, additional, or inverted, with a constrained LLM used solely to produce a review-aid report from the already-computed delta.","pith_inferences":["If the linear-sequence assumption is relaxed to support alt/opt/loop fragments, the same differencing idea could cover a larger share of real automotive sequence diagrams without changing the hardware-trace front end.","The separation of deterministic classification from LLM reporting is a reusable pattern for any safety-critical setting where fabrication risk must stay out of the evidence chain.","The approach could be extended to trend monitoring: successive deltas over releases would surface accumulating drift that single-scenario checks miss."],"forward_implications":["Architectural review can treat the serialized PlantUML delta as primary, inspectable evidence of design-runtime mismatch while the LLM report remains only an aid.","Safety documentation workflows gain an explicit trace from detected message-exchange drift back to commits and work items that introduced the change.","Ordering (inverted) discrepancies remain the hardest class and become the natural next target for method improvement.","The same three-phase pipeline can be inserted into CI/CD so that each new feature-linked test run produces a fresh drift delta for expert review."],"fun_headline_variants":["Runtime traces detect architectural drift in safety firmware","Message diffs flag missing or inverted firmware interactions","Hardware traces catch design drift against sequence diagrams","Deterministic delta classifies firmware architecture discrepancies","Traces turn executions into message sequences to find drift"],"cache_read_input_tokens":128,"weakest_assumption_plain":"Both the design diagrams and the runtime traces must be reducible to the same linear sequence of inter-component messages under a strictly sequential, deterministic execution model; combined fragments and concurrent paths are outside scope.","fun_headline_variants_meta":{"raw":{"variants":["Runtime traces detect architectural drift in safety firmware","Message diffs flag missing or inverted firmware interactions","Hardware traces catch design drift against sequence diagrams","Deterministic delta classifies firmware architecture discrepancies","Traces turn executions into message sequences to find drift"]},"model":"grok-4.5","effort":"low","cost_usd":0.00277,"raw_usage":{"total_tokens":1015,"prompt_tokens":734,"num_sources_used":0,"completion_tokens":50,"cost_in_usd_ticks":27700000,"prompt_tokens_details":{"text_tokens":734,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":231,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":734,"tokens_out":50,"duration_ms":2888,"temperature":1.0,"reasoning_tokens":231,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-12T04:37:33.352195+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"On a held-out set of industrial feature-linked test cases whose design diagrams are linear, measure whether the automatically labeled deltas still reach high F1 against independent expert ground truth for confirmed, missing, and additional classes, and whether practitioners still rate the reports useful when the LLM step is removed or replaced by a non-LLM template.","supporting_citations":[],"review_version":1}