{"id":"d844d7a0-e23d-493f-aa4c-7a134a61ac8a","arxiv_id":"2608.01805","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"CockpitHAT attributes failures in embodied multi-agent cockpit systems by slicing context along dependency distances, adding vehicle and environment evidence, and weighting analyst votes by safety severity, beating text-only baselines on Who&When and reaching 78.3% agent-level accuracy on its own…","lead":"CockpitHAT is a new system that finds which agent and which step caused a failure in multi-agent LLM cockpit assistants, using conversation along with vehicle and environment data. The authors also release CockpitBench, a benchmark of 212 safety-rated failure scenarios for embodied cockpit systems.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Dependency-graph ablation is confounded: Appendix Prompt 1 builds the DAG with a global LLM call, contradicting the §3.1 deterministic pattern-matching claim, so the I2→I3 gains may reflect extra LLM reasoning rather than dependency-distance slicing.","rationale":"The paper's central assertion is that dependency-aware, multi-channel, risk-calibrated attribution outperforms text-only positional attribution, and the experiment designed to prove the dependency component is the I2→I3 ablation. That ablation is only interpretable if the dependency graph is built by the described deterministic matcher. The appendix's Prompt 1 contradicts the method and implementation sections and introduces a global LLM pass. Because all analysts already use the same LLM family, that extra pass could simply add reasoning capacity, making the improvement an artifact of compute or prompt structure rather than of dependency-distance layering. This is an internal inconsistency, not a disagreement with external consensus, so it is directly load-bearing. The reader's weakest assumption about four-signal completeness is related, and the paper's Limitations already concedes it; but the more immediate, testable flaw is the confound: before worrying about missing implicit dependencies, the paper needs to show the graph is built as stated and that slicing, not the graph builder, drives the gains. A 2x2 ablation separating graph-construction method from slicing rule settles this. Other issues (no external baseline on CockpitBench, unused salience score) affect generalization and completeness but do not undercut the internal validity of the central comparison. I keep the conditional verdict.","tokens_in":20457,"tokens_out":10582,"duration_ms":94023,"concrete_test":"Run a 2x2 ablation on Who&When Hand-Crafted and CockpitBench: (A) dependency-distance slicing with a deterministic four-signal matcher and no graph LLM; (B) positional slicing plus an LLM graph-reasoning pass with the same token budget as Prompt 1; (C) dependency-distance slicing using the Appendix Prompt 1 graph; (D) the paper's I2 positional hierarchy. If only C reproduces the reported 0.779/0.378 accuracies, the headline gains come from the extra LLM graph pass, not from dependency-distance slicing. If A matches C, the determinism contradiction is cosmetic but the text should be corrected. Report agent-level and step-exact metrics with DOUBAO-SEED-2.0-PRO analysts.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.1 and §4.1 describe the DAG as produced by deterministic local pattern matching over four operational signals, with only entity-linking delegated to an LLM; the graph is claimed to be deterministic and reusable, with O(n^2) worst-case cost. Appendix A.3, Prompt 1, instead instructs a full LLM to analyze the full trace globally before making local edge decisions and to output the entire edge set with justifications. That is not deterministic pattern matching and is not confined to the four signals. If Prompt 1 was used in the experiments, the central ablation I2→I3 (Table 2) is confounded: I3 changes the slicing rule and simultaneously adds a global graph-reasoning LLM pass that I2 lacks. The reported jumps (+13.7 HC / +8.6 AG agent-level, +25.2 CockpitBench agent-level) could then come from that extra reasoning, not from dependency-distance layering. If Prompt 1 was not used, the paper must state which code path produced Table 2 and why Prompt 1 appears in the methodology appendix. Either way, the claim that dependency-aware slicing is the operative mechanism is not established, and this directly bears on the paper's central conclusion that dependency-aware, multi-channel, risk-calibrated attribution is an effective paradigm.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CockpitHAT, a hierarchical attribution framework for failure diagnosis in embodied multi-agent LLM systems. It constructs a dependency DAG from traces via four operational signals (reference, plan, tool, control), slices each focal step's context by dependency distance rather than positional offset, adds an embodied channel adapter for cockpit signals, and combines a safety-uplifted, confidence-weighted analyst consensus that predicts ISO 26262 ASIL severity. The authors also introduce CockpitBench, 212 annotated traces with ASIL labels and three-expert consensus. On the public Who&When benchmark, CockpitHAT reports agent-level/step-exact accuracies of 77.9%/37.8% on Hand-Crafted and 86.5%/46.0% on Algorithm-Generated, outperforming the ECHO baseline by up to 17.6/16.7 points. On CockpitBench it reports 78.3% agent-level and 38.2% step-exact accuracy. The paper claims this establishes dependency-aware, multi-channel, risk-calibrated attribution as an effective paradigm for reliable failure diagnosis in embodied multi-agent systems.","tokens_in":20729,"tokens_out":6420,"duration_ms":53698,"significance":"The problem is real and timely: Correctness Collapse in safety-critical embodied LLM agents is an important failure mode, and the paper's design of dependency-distance slicing, multi-channel evidence, and ASIL-aware consensus is a plausible direction. CockpitBench, with its ISO 26262 ASIL labels, three-expert consensus, and embodied channel annotations, could be a valuable community resource. The Who&When comparison against ECHO is externally grounded, and the paper reports statistical significance and token costs. However, the central ablation I2→I3 is confounded by the mismatch between the described deterministic graph construction and the appendix's global LLM prompt, and CockpitBench lacks external baseline numbers. The significance of the empirical claims cannot be fully assessed until these issues are resolved; the framework's potential is clear, but the current evidence does not yet establish the mechanism claimed.","major_comments":[{"comment":"The paper states in §3.1 that edge induction 'reduces to local pattern matching plus entity linking' and in §4.1 that graph construction 'uses deterministic pattern matching; only the entity-linking sub-step invokes a lightweight LLM call.' Appendix A.3, Prompt 1, however, instructs the model to 'Analyze the full trace globally before making local edge decisions,' to 'Consider both linguistic and embodied evidence,' and to output the entire edge set with justifications. These descriptions are irreconcilable. If Prompt 1 is the implementation behind Table 2, then the I2→I3 ablation is confounded: I3 adds a global LLM graph-reasoning pass that I2 lacks, so the reported jumps (+13.7 HC, +8.6 AG, +25.2 CockpitBench agent-level) could stem from this extra reasoning rather than from dependency-distance slicing. If Prompt 1 is not the experimental code path, the paper must state which one is and explain why Appendix A.3 presents Prompt 1 as the method. Either way, the central claim that dependency-aware slicing is the operative mechanism is not established. Please provide a controlled ablation in which I3 uses only the deterministic local pattern matcher, or in which I2 is augmented with an equivalent global LLM pass, and clearly state the code path that produced Table 2.","section":"§3.1, §4.1, Appendix A.3 Prompt 1, Table 2"},{"comment":"CockpitBench results are reported only for CockpitHAT in Table 3; no external baseline numbers appear in the main text. Appendix A.1.3 states that 'the state-of-the-art ECHO model cannot achieve saturated performance' but gives no figures, and Table 2's CockpitBench column compares only ablations of CockpitHAT itself. Because CockpitBench is newly introduced by the authors, the absence of independently comparable baselines leaves the embodied-channel contribution unvalidated: the reader cannot determine whether 78.3% agent-level / 38.2% step-exact represents an advance over existing attribution methods on embodied traces. Please report at least ECHO, and ideally the other Who&When baselines, on CockpitBench under the same evaluation protocol.","section":"§4.3, Table 3, Appendix A.1.3"},{"comment":"The additive ablation chain I1→I6 is central to the paper, but the operational definitions of the intermediate variants are not given. In particular, the paper never states what 'Objective Analysis (I4)' adds over I3, nor what 'Decoupled Attribution (I5)' changes relative to I4, beyond the prose labels in Section 3.4. Without a precise mapping from each I_k to a concrete pipeline configuration, the ablation is not reproducible, and the attribution of gains to specific components (e.g., 'Decoupled Attribution ... contributes the table's largest single AG agent-level jump, +17.6 points') cannot be verified. Please specify the exact configuration of each ablation variant, including which modules are enabled or disabled.","section":"§4.1, Table 2, §3.4"},{"comment":"Several hyperparameters required to reproduce the method are not reported with values. The disagreement threshold δ_cons(η*) in §3.4 is described only as 'monotone in severity'; no values or calibration procedure are given. The look-back cap L used in Algorithm 1 is not specified anywhere in the paper. Also, the safety-uplift parameters λ=0.5, β=0.4, κ=1.0 are acknowledged as hand-calibrated for the cockpit domain in the Limitations, which is acceptable if clearly flagged, but the paper applies this uplift on Who&When, where no ASIL ground truth exists, without discussing how the predicted severities are grounded. Please provide δ_cons and L, state the calibration source for λ/β/κ, and comment on the validity of applying the cockpit-calibrated uplift to non-cockpit traces.","section":"§4.1, Algorithm 1, Eqs. (3)-(4)"}],"minor_comments":[{"comment":"There is a stray 'onG, , causing' typo in the sentence about the slicing rule compressing seeded steps.","section":"§3.3"},{"comment":"The phrase 'surpassing the text-only SOTA ECHO by up to 17.6 / 16.7 points' is ambiguous; please specify that the first number is agent-level and the second is step-level accuracy.","section":"Abstract and §1"},{"comment":"The P-values in the backbone ablation table lack a footnote describing the statistical test; please state whether these are the same chi-squared tests as in Table 1 or a different procedure.","section":"Table 6"},{"comment":"The statement 'permissive open-source license' is vague; specifying the exact license (e.g., Apache 2.0) would help readers assess reuse conditions.","section":"Ethics Statement"}],"recommendation":"major_revision","confidential_remarks":"The discrepancy between the deterministic pattern-matching claim in Sections 3.1/4.1 and the global LLM prompt in Appendix A.3 is the most serious issue in this manuscript. It directly undermines the paper's principal ablation and, depending on the authors' response, could require rerunning the central experiments. I would weigh that response heavily: if the Table 2 results came from a global LLM graph-construction path, the current claims about dependency-distance slicing are not supported. The lack of external baselines on CockpitBench is a secondary but important gap that should be addressed before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Candid take: this is a real contribution, a new domain (embodied cockpit attribution) and the first benchmark (CockpitBench) that ties attribution to embodied channels and ASIL safety labels. The Who&When evaluation is credible: re-implemented baselines, chi-squared tests, tolerance curves, token costs. The authors also do the field a service by openly listing limitations (four-signal dependency graph, hand-tuned uplift, post-hoc only). CockpitBench's three-expert annotation with inter-rater stats and independent safety audit for ASIL-C/D is above the usual bar for this area.\n\nThe soft spots are mostly about what isn't shown. There are no external baselines on CockpitBench, no code/data release yet, and the safety-uplift hyperparameters are acknowledged as cockpit-tuned. That alone would make me want a revised version rather than a desk rejection. But there is a sharper problem. Section 3.1 says the DAG comes from deterministic pattern matching plus entity linking, with claimed O(n^2) worst-case cost. Appendix A.3, Prompt 1, tells a different story: the model must \"analyze the full trace globally before making local edge decisions\" and output the whole edge set with justifications. If that prompt was used in the experiments, the Table 2 ablation I2->I3 is confounded: you changed the context slicing and simultaneously added a global graph-reasoning step that I2 lacks. The +13.7/+8.6/+25.2 agent-level jumps could come from that extra pass rather than from dependency-distance layering. If Prompt 1 was not used, the paper must state which code path produced Table 2 and why the prompt appears in the appendix. The Who&When headline results survive this intact, because they compare full CockpitHAT against ECHO, not the isolated slicing rule; but the abstract's \"effective paradigm\" claim is specifically about dependency-awareness, and that claim is not established until the ablation is clean. This is fixable, but it has to be fixed.\n\nOne more minor point: the salience score p_i from Equation 1 is computed but never consumed; the algorithm uses only the tags and the forced promotion. Leftover or unused, it should be cut or made functional.\n\nBottom line: worth a serious referee. I'd send it out and ask for a clean ablation, an explicit statement about Prompt 1, and ideally release of the benchmark and code. If the authors deliver that, this becomes a useful resource for anyone debugging embodied multi-agent systems.","headline":"Solid new benchmark and credible external results, but the dependency-graph ablation is confounded by a global-LLM prompt that contradicts the claimed deterministic pipeline.","tokens_in":21290,"tokens_out":2812,"would_cite":false,"duration_ms":26419,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"For embodied multi-agent failures, dependency distance—not transcript position—should drive blame attribution.","keywords":["LLM multi-agent systems","error attribution","dependency graph","hierarchical context","embodied safety-critical agents","ISO 26262 ASIL","CockpitBench","Correctness Collapse"],"falsifier":"Take a set of traces whose true error depends on an implicit shared fact (for example, two agents both act on an unstated map constraint). Run CockpitHAT on those traces and check step-exact accuracy. The four-signal graph cannot draw an edge through shared background knowledge, so if the method still localizes those failures at chance level, the dependency-graph premise is falsified; if it localizes them anyway, the LLM entity linker or another component is carrying the weight.","tokens_in":20230,"feed_emoji":"🚗","tokens_out":9510,"duration_ms":80127,"temperature":0.7,"pith_summary":"LLM multi-agent systems can look accurate while failing badly at the process level, a pattern the paper calls Correctness Collapse, and in a car cockpit a lexically correct utterance can trigger a dangerous physical action. The paper's central claim is that finding which agent and which step caused a failure should be driven by dependency structure rather than by how close steps appear in the transcript. It introduces CockpitHAT, which lifts each trace into an interaction DAG using reference, plan, tool, and control signals, slices per-step context by dependency distance, attaches embodied-channel evidence, and combines analyst votes with a safety-weighted consensus. On the public Who&When benchmark, the paper reports 77.9 and 86.5 percent agent-level accuracy and 37.8 and 46.0 percent step-exact accuracy, gains over the text-only ECHO baseline of up to 17.6 and 16.7 points. It also releases CockpitBench, 212 failure traces labeled with ISO 26262 ASIL severity and three-channel embodied evidence.","feed_headline":"Dependency distance beats text position for multi-agent blame","feed_subtitle":"CockpitHAT reaches 77.9–86.5 percent agent attribution on Who&When by slicing context along dependency edges.","key_machinery":"The load-bearing object is the interaction DAG with its dependency-distance slicing. Each trace step becomes a node, and an edge vi→vj is added when step j depends on step i through one of four signals: an explicit reference to an entity or tool result, execution or refinement of a proposed plan, use of a tool output field, or dispatch/control invocation. Dependency distance is the shortest undirected path between nodes in that graph, and each focal step is presented to the analysts as four layers (immediate, local, distant, global) determined by that distance rather than by transcript position. The second mechanism is the Embodied Channel Adapter, a lightweight per-trace scan that tags steps as interaction-risk, environmental misgrounding, safety-violation, or cross-domain and promotes them one layer (safety violations into the immediate layer) with matching channel fields attached. The third is the safety-uplifted consensus: a mandatory Safety Analyst plus three persona-sampled analysts, a maximum-severity prior over ASIL levels, and an exponential uplift on the safety analyst's vote at high severity, with a disagreement-resolution fallback when margins are thin. These three mechanisms form the argument that error propagation travels along dependency edges, so context, evidence, and voting all need to be dependency- and risk-aware.","core_discovery":"CockpitHAT's central discovery is that dependency-aware context outperforms position-aware context for locating the responsible agent and step. Instead of taking the k nearest transcript positions around a focal step, the method builds a directed acyclic graph whose edges mark reference, plan, tool, and control dependencies, then defines dependency distance as the shortest undirected path in that graph and uses it to build four context layers: verbatim immediate neighbors, compressed local, summarized distant, and milestone global. An embodied adapter promotes flagged steps upward, forcing safety violations into the immediate layer regardless of distance. A four-analyst panel then reaches a two-stage consensus in which severity is chosen by the maximum ASIL level among confident analysts and the safety analyst's vote is exponentially uplifted at high severity. On CockpitBench the method reaches 78.3 percent agent-level and 38.2 percent step-exact accuracy, and it attributes high-severity failures (ASIL C/D) more accurately than low-severity ones. The paper's case is that these results establish dependency-aware, multi-channel, risk-calibrated attribution as the right paradigm for diagnosing failures in embodied multi-agent systems.","pith_inferences":["As an extension, replacing the four hand-specified edge signals with learned or provenance-based dependency labels would directly test whether the dependency graph, rather than the voting stage, is the performance ceiling; better edge recall should raise step-exact accuracy.","As an extension, the same dependency-distance slicing could transfer to code-level or hardware-in-the-loop multi-agent systems where a static call graph already exists, turning attribution into a graph-metrics problem.","As an extension, the fixed automotive safety-uplift parameters could be made adaptive to a domain's ASIL distribution; the paper's own limitation note indicates this is an open knob.","As an extension, because CockpitBench has only 26 ASIL-D traces, the reported high-severity step accuracy would need a larger severity-stratified sample before the risk-calibrated advantage is treated as settled."],"forward_implications":["Attribution pipelines for multi-agent systems should replace positional context windows with dependency-distance layers, because error propagation follows dependency edges rather than transcript adjacency.","Embodied failure diagnosis needs multi-channel evidence: dialogue alone cannot see a lexically correct command that triggers a hazardous vehicle-state change.","Safety-severity labels should be part of attribution benchmarks, since methods can be accurate overall while degrading on the rare high-severity cases that matter most.","A safety-uplifted consensus prevents high-risk failures from being outvoted by generic analyst signals, at a modest token cost (68,287 tokens per trace on the Hand-Crafted split)."],"supporting_citations":[{"why":"Defines the ECHO hierarchical positional-consensus baseline that CockpitHAT replaces, and supplies the two Who&When evaluation splits.","marker":"Banerjee et al., 2025"},{"why":"Formalizes the which-agent/which-step attribution problem and the Who&When benchmark, giving the task and ground-truth labels.","marker":"Zhang et al., 2025"},{"why":"Provides the AgentRx failure taxonomy and trajectory-diagnosis pipeline that motivates multi-perspective attribution and benchmark gaps.","marker":"Barke et al., 2026"},{"why":"Introduces the Correctness Collapse phenomenon that motivates process-level failure diagnosis.","marker":"Yang et al., 2025"},{"why":"Defines ISO 26262 ASIL A-D severity levels used for CockpitBench labels and the safety-uplifted consensus.","marker":"International Organization for Standardization, 2018"},{"why":"Gives the Vehicle Signal Specification used for the vehicle-state channel and 100 ms state snapshots in CockpitBench.","marker":"COVESA, 2024"},{"why":"Provides MP-Bench, cited as a text-only, safety-free attribution benchmark whose limitations justify CockpitBench's design.","marker":"In et al., 2026"}],"fun_headline_variants":["Dependency graph beats position for multi-agent blame","CockpitHAT: attribute failures by dependency, not text position","Safety-correct attribution: dependency distance wins","Why dependency edges trump text windows in agent blame","Multi-agent failure diagnosis: dependency-aware attribution"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the interaction DAG built from the four explicit signals—reference, plan, tool, control—plus entity linking captures every dependency that matters; any causal link that is implicit or based on shared background knowledge will be missing from the graph and can push the true error step into a compressed layer.","fun_headline_variants_meta":{"raw":{"variants":["Dependency graph beats position for multi-agent blame","CockpitHAT: attribute failures by dependency, not text position","Safety-correct attribution: dependency distance wins","Why dependency edges trump text windows in agent blame","Multi-agent failure diagnosis: dependency-aware attribution"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000168,"raw_usage":{"total_tokens":1321,"prompt_tokens":1065,"completion_tokens":256,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":681,"completion_tokens_details":{"reasoning_tokens":182}},"tokens_in":681,"tokens_out":256,"duration_ms":2995,"temperature":1.0,"reasoning_tokens":182,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T15:04:33.843697+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a set of traces whose true error depends on an implicit shared fact (for example, two agents both act on an unstated map constraint). Run CockpitHAT on those traces and check step-exact accuracy. The four-signal graph cannot draw an edge through shared background knowledge, so if the method still localizes those failures at chance level, the dependency-graph premise is falsified; if it localizes them anyway, the LLM entity linker or another component is carrying the weight.","supporting_citations":[],"review_version":2}