pith. sign in

arxiv: 2606.24546 · v1 · pith:QE43INCDnew · submitted 2026-06-23 · 💻 cs.RO

Explaining Failures of Cyber-Physical Systems with Actual Causality

Khen Elimelech , Tom Yaacov , David A. Kelly , Hana Chockler , Moshe Y. Vardi This is my paper

Pith reviewed 2026-06-25 23:47 UTC · model grok-4.3

classification 💻 cs.RO
keywords actual causalitycyber-physical systemsfailure explanationautonomous vehiclesneural networksexplanation derivation
0
0 comments X

The pith

Actual causality can explain failures in cyber-physical systems like autonomous cars despite black-box components.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper establishes that the actual causality framework, previously applied only to simple systems such as image classifiers, can be extended to derive explanations for failures in complex cyber-physical systems. A sympathetic reader would care because these systems cannot be fully verified before deployment due to neural components, making unexpected failures inevitable and explanations essential for building trust and enabling mitigation. The work addresses the theoretical gaps required for correct application in the CPS domain and supplies two system-agnostic algorithms that prioritize either explanation optimality or derivation efficiency. The approach is shown on a neural-network-controlled car that avoids collisions.

Core claim

The paper claims that actual causality can be leveraged for CPS failure explanation once theoretical gaps are closed, and supplies two practical algorithms to generate such explanations in a system-agnostic way.

What carries the argument

The actual causality framework, which identifies causes of specific outcomes via counterfactual interventions, now applied to CPS failure explanation.

If this is right

  • Explanations for CPS failures become possible without full prior verification of system behavior.
  • Users can select between algorithms that favor optimal explanations or faster computation.
  • The method works on neural-network-controlled vehicles for tasks such as collision avoidance.
  • Derived explanations support improved trust and post-failure mitigation steps.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same extension could apply to other autonomous platforms such as delivery drones.
  • Pairing the method with runtime monitors might allow on-the-fly failure diagnosis during operation.
  • Regulators could require such causal explanations as part of safety audits for deployed CPS.

Load-bearing premise

Black-box neural components in CPS can be modeled sufficiently for actual causality analysis once the theoretical gaps are addressed.

What would settle it

A run of the algorithm on the autonomous car example that produces an explanation contradicting the known collision-avoidance logic or expert reconstruction of the failure.

Figures

Figures reproduced from arXiv: 2606.24546 by David A. Kelly, Hana Chockler, Khen Elimelech, Moshe Y. Vardi, Tom Yaacov.

Figure 1
Figure 1. Figure 1: Our running example (as introduced in [20]): an [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: The CPS testing model and the corresponding causal model for the CPS simulation. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Two simulation runs demonstrate our insights: (1) [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Median explanation effort required by ES, and the [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
read the original abstract

Modern autonomous Cyber-Physical Systems (CPSs), such as self-driving cars, face increasingly complex demands, and yet are expected to act reliably. The black-box nature often characterizing such systems, especially those relying on neural components, makes it impossible to fully verify the system behavior prior to deployment. Unfortunately, unexpected failures-when the system does not comply with its specification-are inevitable and may have catastrophic implications. To improve trust in the system and facilitate future mitigation after a failure occurs, it is important to try to derive an explanation for the unexpected system behavior. This paper introduces the novel concept of leveraging the framework of actual causality for CPS failure explanation. Up until now, this framework was only used to derive explanations in the context of simple systems, such as image classifiers. This paper addresses the theoretical gaps and provides the guidance needed to allow for correct explanation derivation in the CPS domain. Beyond the theoretical contribution, the paper presents two novel, practical, system-agnostic explanation derivation algorithms, allowing to prioritize either explanation optimality or derivation efficiency. The approach is demonstrated and evaluated in the context of a neural-network-controlled autonomous car, designed to avoid collisions.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 2 minor

Summary. The paper claims to extend the actual causality (Halpern-Pearl) framework to explain failures in cyber-physical systems containing black-box neural components. It asserts that prior applications were limited to simple systems like image classifiers, identifies theoretical gaps for the CPS domain, supplies guidance to close them, and introduces two system-agnostic algorithms (one prioritizing optimality, one efficiency) that are evaluated on a neural-network-controlled autonomous vehicle tasked with collision avoidance.

Significance. If the claimed guidance permits well-defined interventions and counterfactuals on black-box neural controllers without requiring white-box access or exhaustive prior verification, the work would offer a practical route to post-failure explanation in autonomous CPS where full verification is infeasible. The provision of two concrete algorithms and an empirical demonstration on a vehicle controller would strengthen the contribution.

major comments (3)
  1. [§4] §4 (Modeling CPS for Actual Causality): the guidance for constructing a structural causal model does not specify how endogenous variables and structural equations are obtained for neural-network outputs when only black-box access is available; without this, the interventions required by the Halpern-Pearl definition remain undefined.
  2. [§5.2–5.3] §5.2–5.3 (Algorithm descriptions): both algorithms presuppose an already-faithful SCM that supports counterfactual queries; this assumption is load-bearing for the central claim yet is not discharged by the supplied theoretical guidance, leaving the algorithms inapplicable under the paper’s own premise that full verification is impossible.
  3. [§6] §6 (Evaluation on autonomous car): the reported explanations rely on an implicit simulator model whose fidelity to the deployed black-box controller is not quantified; this undermines the claim that the method works for unverifiable neural CPS.
minor comments (2)
  1. [§3] Notation for continuous-state variables and intervention operators is introduced without a consolidated table; a single reference table would improve readability.
  2. [Abstract] The abstract states that the framework was previously limited to “simple systems such as image classifiers,” but does not cite the specific prior works; adding those references would clarify the novelty claim.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the constructive comments, which help clarify the scope and limitations of applying actual causality to black-box CPS. We respond to each major comment below and indicate where revisions will be made to strengthen the manuscript.

read point-by-point responses

  1. Referee: [§4] §4 (Modeling CPS for Actual Causality): the guidance for constructing a structural causal model does not specify how endogenous variables and structural equations are obtained for neural-network outputs when only black-box access is available; without this, the interventions required by the Halpern-Pearl definition remain undefined.

    Authors: Section 4 defines endogenous variables to include neural outputs and structural equations as the input-output mapping realized by the network. For black-box access, the equation is the observable function of the network; interventions are performed by setting input variables and querying the network to obtain the resulting output value. This supports the required counterfactuals without internal access. We will revise §4 to include an explicit example of this modeling for a neural controller and clarify the intervention mechanism. revision: yes

  2. Referee: [§5.2–5.3] §5.2–5.3 (Algorithm descriptions): both algorithms presuppose an already-faithful SCM that supports counterfactual queries; this assumption is load-bearing for the central claim yet is not discharged by the supplied theoretical guidance, leaving the algorithms inapplicable under the paper’s own premise that full verification is impossible.

    Authors: The algorithms operate on an SCM constructed per the §4 guidance, where faithfulness is achieved by accurately capturing known dynamics and treating the neural component via its observable mapping. Counterfactual queries are realized by intervening on inputs and re-evaluating (including network queries with modified inputs). We agree that the link between modeling and algorithms needs to be more explicit and will revise §5 to reference the black-box handling from §4 and discuss applicability when full verification is unavailable. revision: yes

  3. Referee: [§6] §6 (Evaluation on autonomous car): the reported explanations rely on an implicit simulator model whose fidelity to the deployed black-box controller is not quantified; this undermines the claim that the method works for unverifiable neural CPS.

    Authors: The evaluation employs a simulator that executes the identical neural controller code as the target system. We did not report quantitative fidelity metrics. We will add discussion in §6 on the simulator construction, its equivalence to the deployed controller, and any resulting limitations for the explanations. revision: yes

Circularity Check

0 steps flagged

No circularity: adaptation of external actual-causality framework with no self-referential reduction.

full rationale

The provided abstract and description present the work as an extension of the pre-existing Halpern-Pearl actual causality framework to CPS, explicitly noting that the framework had previously been applied only to simpler systems. No equations, algorithms, or claims are shown to reduce by construction to their own inputs, fitted parameters renamed as predictions, or load-bearing self-citations. The two system-agnostic algorithms are described as novel outputs rather than tautological restatements. The central premise relies on external theoretical foundations and does not exhibit any of the enumerated circular patterns.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract provides no explicit free parameters, axioms, or invented entities; the work relies on the pre-existing actual causality framework plus unspecified modeling assumptions for CPS.

pith-pipeline@v0.9.1-grok · 5744 in / 986 out tokens · 24073 ms · 2026-06-25T23:47:09.230815+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

20 extracted references · 1 linked inside Pith

  1. [1]

    Toward verified artificial intelligence,

    S. A. Seshia, D. Sadigh, and S. S. Sastry, “Toward verified artificial intelligence,”Commun. ACM, vol. 65, no. 7, p. 46–55, June 2022

    2022

  2. [2]

    Stream- lined integration of gr(1) synthesis and reinforcement learning for optimizing critical cyber-physical systems,

    E. Wete, J. Greenyer, T. Yaacov, D. Kudenko, and W. Nejdl, “Stream- lined integration of gr(1) synthesis and reinforcement learning for optimizing critical cyber-physical systems,” in2025 ACM/IEEE 28th International Conference on Model Driven Engineering Languages and Systems (MODELS), 2025, pp. 36–47

    2025

  3. [3]

    A Survey of Algorithms for Black-Box Safety Validation of Cyber-Physical Systems,

    A. Corso, R. Moss, M. Koren, R. Lee, and M. Kochenderfer, “A Survey of Algorithms for Black-Box Safety Validation of Cyber-Physical Systems,”Journal of Artificial Intelligence Research, vol. 72, pp. 377– 428, Oct. 2021

    2021

  4. [4]

    Causes and explanations: A structural- model approach. Part I: Causes,

    J. Y . Halpern and J. Pearl, “Causes and explanations: A structural- model approach. Part I: Causes,”British Journal for the Philosophy of Science, vol. 56, no. 4, 2005

    2005

  5. [5]

    Pearl,Causality

    J. Pearl,Causality. Cambridge university press, 2009

    2009

  6. [6]

    J. Y . Halpern,Actual Causality. The MIT Press, 2019

    2019

  7. [7]

    Explaining image classifiers,

    H. Chockler and J. Y . Halpern, “Explaining image classifiers,” in Proceedings of the 21st International Conference on Principles of Knowledge Representation and Reasoning, KR, 2024

    2024

  8. [9]

    Hume,A Treatise of Human Nature

    D. Hume,A Treatise of Human Nature. John Noon, 1739

  9. [10]

    Causation,

    D. K. Lewis, “Causation,”Journal of Philosophy, vol. 70, pp. 556– 567, 1973

    1973

  10. [11]

    Causal explana- tions for image classifiers,

    H. Chockler, D. A. Kelly, D. Kroening, and Y . Sun, “Causal explana- tions for image classifiers,”arXiv preprint arXiv:2411.08875, 2024

    Pith/arXiv arXiv 2024

  11. [12]

    Multiple different explanations for image classifiers,

    H. Chockler, D. A. Kelly, and D. Kroening, “Multiple different explanations for image classifiers,” inECAI European Conference on Artificial Intelligence, 2025

    2025

  12. [13]

    I am big, you are little; i am right, you are wrong,

    D. A. Kelly, A. Chanchal, and N. Blake, “I am big, you are little; i am right, you are wrong,” inIEEE/CVF International Conference on Computer Vision, ICCV. IEEE, 2025

    2025

  13. [14]

    Responsibility and blame: A structural-model approach,

    H. Chockler and J. Y . Halpern, “Responsibility and blame: A structural-model approach,”J. Artif. Intell. Res., vol. 22, pp. 93–115, 2004

    2004

  14. [15]

    Causality for cyber-physical systems,

    H. Araujo, H. Chockler, M. R. Mousavi, G. Carvalho, and A. Sampaio, “Causality for cyber-physical systems,”arXiv preprint arXiv:2505.13475, 2025

    arXiv 2025

  15. [16]

    Why did I fail? A causal-based method to find explanations for robot failures,

    M. Diehl and K. Ramirez-Amaro, “Why did I fail? A causal-based method to find explanations for robot failures,”IEEE Robotics and Automation Letters, vol. 7, no. 4, pp. 8925–8932, 2022

    2022

  16. [17]

    Analyzing neighborhoods of falsifying traces in cyber-physical systems,

    R. D. Diwakaran, S. Sankaranarayanan, and A. Trivedi, “Analyzing neighborhoods of falsifying traces in cyber-physical systems,” ser. ICCPS ’17. New York, NY , USA: Association for Computing Machinery, 2017, p. 109–119

    2017

  17. [18]

    Automatic failure explanation in cps models,

    E. Bartocci, N. Manjunath, L. Mariani, C. Mateis, and D. Ni ˇckovi´c, “Automatic failure explanation in cps models,” inSoftware Engi- neering and Formal Methods: 17th International Conference, SEFM 2019, Oslo, Norway, September 18–20, 2019, Proceedings. Berlin, Heidelberg: Springer-Verlag, 2019, p. 69–86

    2019

  18. [19]

    Faultex: Explaining operational changes in terms of design variables in cps control code,

    A. Banerjee, I. Lamrani, and S. K. Gupta, “Faultex: Explaining operational changes in terms of design variables in cps control code,” in2021 4th IEEE International Conference on Industrial Cyber- Physical Systems (ICPS), 2021, pp. 485–490

    2021

  19. [20]

    Falsification of autonomous systems in rich environments,

    K. Elimelech, M. Lahijanian, L. E. Kavraki, and M. Y . Vardi, “Falsification of autonomous systems in rich environments,”ACM Transactions on Cyber-Physical Systems (TCPS), 2026, to appear

    2026

  20. [21]

    LiteRacer: a lightweight autonomous vehicle simulator for benchmarking and development of formal verification techniques,

    ——, “LiteRacer: a lightweight autonomous vehicle simulator for benchmarking and development of formal verification techniques,” in Workshop on Software Challenges in Formal Methods for Robotics (FMR), in conjunction with IEEE International Conference on Robotics and Automation (ICRA), 05 2024. [Online]. Available: https://github.com/khen/LiteRacer

    2024