pith. sign in

arxiv: 2509.07674 · v2 · pith:W2YAAHOWnew · submitted 2025-09-09 · 💻 cs.RO · cs.HC

Temporal Counterfactual Explanations of Behaviour Tree Decisions

Tamlin Love , Antonio Andriella , Guillem Aleny\`a This is my paper

Pith reviewed 2026-05-21 21:49 UTC · model grok-4.3

classification 💻 cs.RO cs.HC
keywords counterfactual explanationsbehavior treesrobot explainabilitycausal modelsdecision makingroboticstemporal explanations
0
0 comments X

The pith

A causal model built automatically from a behavior tree's structure and domain knowledge generates diverse counterfactual explanations for robot decisions in real time.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces a method for behavior-tree-controlled robots to answer contrastive why questions using causal counterfactual explanations. It works by first constructing a causal model directly from the tree's structure together with supplied knowledge about states and individual nodes. The model is then queried and searched to produce a set of diverse counterfactuals that account for the observed decision. The resulting explanations are shown to be consistent and accurate across many tree structures and states while running fast enough for live use, addressing gaps left by earlier techniques that could not deliver causal contrastive answers.

Core claim

By first automatically building a causal model from the structure of the behaviour tree as well as domain knowledge about the state and individual behaviour tree nodes, the resultant causal model is then queried and searched to find a set of diverse counterfactual explanations that correctly explain the behaviour of a wide range of behaviour tree structures and states in real time.

What carries the argument

The automatically constructed causal model from the behavior tree structure plus domain knowledge about states and nodes, which is queried and searched to produce diverse counterfactual explanations.

If this is right

  • Robots can answer contrastive why questions with causal rather than merely descriptive explanations.
  • Explanations remain consistent with the underlying tree logic across varied structures and states.
  • Generation occurs fast enough to support real-time human-robot interaction.
  • A single method covers a broad range of behavior-tree designs without manual redesign of the explainer.
  • Greater transparency supports safer and more trustworthy deployment of behavior-tree robots.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same causal-model construction could be applied to other hierarchical control structures beyond behavior trees.
  • Users could query explanations interactively to anticipate or steer future robot actions.
  • Physical robot trials with human participants would test whether the generated counterfactuals improve user understanding.
  • Combining the method with natural-language interfaces would let non-experts pose why questions directly.

Load-bearing premise

The causal model built from the behavior tree structure and supplied domain knowledge accurately and completely captures the decision logic and all relevant state dependencies.

What would settle it

A case in which a generated counterfactual, when applied to the behavior tree, produces a different decision outcome than the tree actually executes under the modified state.

Figures

Figures reproduced from arXiv: 2509.07674 by Antonio Andriella, Guillem Aleny\`a, Tamlin Love.

Figure 1
Figure 1. Figure 1: A diagram depicting the proposed architecture for explaining BT execution. At runtime, snapshots of the BT and the state are saved in an episodic memory (Section 5.1). Our contributions are twofold. The first contribution is an algorithm that builds a causal model (the explanation model) from the structure of the BT and state and decision-making knowledge (Section 5.2). Our second contribution is an algori… view at source ↗
Figure 2
Figure 2. Figure 2: Explanation subgraphs of 𝑖 . For leaf nodes, 𝐸𝑖 , 𝑑𝑖 and 𝑟𝑖 are related as in (a). For composite nodes, 𝑟𝑖 is related to 𝐸𝑖 and 𝑟𝑐 1 ,...,𝑟𝑐𝑁 as in (b), where 𝑐 1 , ..., 𝑐𝑁 ∈ 𝐶ℎ(𝑖 ). When 𝑖 is the left-most child of its parent 𝑝 = 𝑃 𝑎(𝑖 ), 𝐸𝑖 is dependent on 𝐸𝑝 , as in (c). Otherwise, 𝐸𝑖 is dependent on 𝑟𝑗 , as in (d), where 𝑗 is the sibling of 𝑖 immediately to its left. First of all, we note that… view at source ↗
Figure 3
Figure 3. Figure 3: An example of the edges added by Algorithm 3 for one 𝐿𝑖 . In this example, the state graph  is identical to the one in [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: An example state graph with four variables in . 5.2.2. Incorporating Domain Knowledge Once the partial explanation graph has been constructed from the known BT structure, further causal relationships can be inserted from domain knowledge. This process is depicted in Algorithm 3, and an example is provided in [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: An example BT with a fallback node ? , sequence node →, condition node 𝐿0 and action nodes 𝐿1 and 𝐿2 . The leaf nodes in this tree have the following input, output and action sets: 0 = {𝑋𝑎 }, 1 = {𝑋𝑎 , 𝑋𝑐 }, 2 = {𝑋𝑑 }, 0 = ∅, 1 = {𝑋𝑏 } = 2 , 0 = ∅, 1 = {𝑎0 , 𝑎1 } , 2 = {𝑎2 , 𝑎3 } [PITH_FULL_IMAGE:figures/full_fig_p013_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: The complete explanation model for the behaviour tree in [PITH_FULL_IMAGE:figures/full_fig_p013_6.png] view at source ↗
read the original abstract

Explainability, in particular, the ability for robots to explain why they have made a decision or behaved in a certain way, is a critical tool in helping users understand the robots they interact and coexist with. Behaviour trees are a popular framework for controlling the decision-making of robots, and thus a natural question to ask is whether or not a system driven by a behaviour tree is capable of answering "why" questions. While explainability for behaviour tree-driven robots has seen some prior attention, no existing methods are capable of generating causal, counterfactual explanations which detail the reasons for robot decisions and behaviour. Therefore, in this work, we introduce a novel approach which automatically generates counterfactual explanations in response to contrastive "why" questions. Our method achieves this by first automatically building a causal model from the structure of the behaviour tree as well as domain knowledge about the state and individual behaviour tree nodes. The resultant causal model is then queried and searched to find a set of diverse counterfactual explanations. We demonstrate that our approach is able to correctly explain the behaviour of a wide range of behaviour tree structures and states in real time, unlike previous methods which are either unable to answer contrastive questions with causal explanations, or are not guaranteed to provide consistent and accurate explanations. By being able to answer a wide range of causal queries, our approach represents a step towards more transparent, understandable, and ultimately safe and trustworthy robotic systems.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The paper introduces a method for generating temporal counterfactual explanations for decisions made by robots controlled via Behaviour Trees (BTs). The approach first automatically constructs a causal model from the BT structure combined with supplied domain knowledge about states and node behaviors, then queries and searches this model to produce diverse counterfactuals in response to contrastive 'why' questions. The authors claim the method correctly explains a wide range of BT structures and states in real time, offering causal consistency that prior methods lack.

Significance. If the causal model construction faithfully encodes BT execution semantics including temporal ordering and node-type dependencies, the work would provide a valuable advance in robot explainability, supporting safer and more transparent human-robot systems. The emphasis on automatically deriving causal structure from BTs plus domain knowledge, combined with diversity in explanations, represents a targeted contribution to contrastive reasoning in robotics; reproducible examples or code would strengthen this.

major comments (2)
  1. [Abstract / Causal model construction] Abstract and method description: the central claim that the automatically built causal model produces explanations 'consistent and accurate' with actual BT behaviour rests on the unverified assumption that the construction process fully captures temporal execution semantics (tick ordering, state persistence across ticks, Sequence/Selector/Parallel node logic, and memory nodes). No explicit mapping rules, soundness argument, or handling of implicit dependencies are provided, so counterfactual queries may be internally consistent within the model yet diverge from real robot execution.
  2. [Abstract / Results] Evaluation claims: the assertion that the method 'correctly explain[s] the behaviour of a wide range of behaviour tree structures and states in real time' is stated without quantitative metrics, error rates, success criteria, or comparison baselines. This undermines the empirical support for the real-time and correctness guarantees.
minor comments (2)
  1. [Method] Notation for causal variables and interventions should be introduced with explicit definitions early in the method section to improve readability.
  2. [Abstract] The abstract mentions 'temporal' counterfactuals but the provided description does not clarify how time is represented in the causal graph (e.g., via time-indexed nodes or explicit ordering constraints).

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their constructive comments, which have helped us identify areas where the manuscript can be strengthened. We address each major comment in detail below and outline the revisions we will make.

read point-by-point responses

  1. Referee: [Abstract / Causal model construction] Abstract and method description: the central claim that the automatically built causal model produces explanations 'consistent and accurate' with actual BT behaviour rests on the unverified assumption that the construction process fully captures temporal execution semantics (tick ordering, state persistence across ticks, Sequence/Selector/Parallel node logic, and memory nodes). No explicit mapping rules, soundness argument, or handling of implicit dependencies are provided, so counterfactual queries may be internally consistent within the model yet diverge from real robot execution.

    Authors: We agree that the current description of causal model construction would benefit from greater explicitness. The manuscript outlines how the causal model is derived from BT structure and domain knowledge, but we acknowledge the absence of detailed mapping rules for temporal aspects and a dedicated soundness argument. In the revised manuscript we will add a new subsection that provides explicit mapping rules for each node type (including how Sequence and Selector enforce ordering via directed edges, how memory nodes persist state across ticks, and how Parallel nodes handle concurrent dependencies). We will also include a soundness argument showing that the resulting causal graph preserves the execution semantics of standard BTs, ensuring that counterfactual interventions correspond to valid alternative execution traces. This revision will directly address the concern that queries could diverge from real robot behaviour. revision: yes

  2. Referee: [Abstract / Results] Evaluation claims: the assertion that the method 'correctly explain[s] the behaviour of a wide range of behaviour tree structures and states in real time' is stated without quantitative metrics, error rates, success criteria, or comparison baselines. This undermines the empirical support for the real-time and correctness guarantees.

    Authors: The evaluation section currently relies on a series of illustrative case studies covering diverse BT structures to demonstrate correctness and real-time performance. We recognise that these qualitative demonstrations would be strengthened by quantitative evidence. In the revision we will augment the evaluation with quantitative metrics: average query runtime across a benchmark set of BTs, success rate of generated counterfactuals when validated against a BT simulator, and a comparison of explanation diversity and consistency against prior non-causal BT explanation methods. These additions will provide clearer empirical support for the claims of correctness and real-time operation. revision: yes

Circularity Check

0 steps flagged

No circularity: causal model is constructed from BT structure plus external domain knowledge

full rationale

The paper's central method is a constructive procedure: it builds a causal model directly from the observable structure of the behaviour tree together with separately supplied domain knowledge about states and node semantics, then performs queries and searches on that model to produce counterfactuals. No equations, definitions, or steps in the abstract or described approach reduce a claimed result to a fitted parameter or to the target explanation by construction. No load-bearing self-citation chain is invoked to justify soundness or uniqueness; the derivation remains independent of the outputs it generates. This is the normal case of a self-contained algorithmic construction.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Review performed on abstract only; full technical details of model construction, search procedure, and any modeling assumptions are unavailable. No free parameters, axioms, or invented entities can be identified from the provided text.

pith-pipeline@v0.9.0 · 5782 in / 1247 out tokens · 38785 ms · 2026-05-21T21:49:26.445149+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

24 extracted references · 24 canonical work pages · 1 internal anchor

  1. [1]

    Relation-based counterfactual explanations for Bayesian network classifiers, in: International Joint Conference on Artificial Intelligence, pp. 451–457. doi:10.5555/3491440.3491503. Andriella, A., Torras, C., Alenyà, G.,

    work page doi:10.5555/3491440.3491503

  2. [2]

    Learning robot policies using a high-level abstraction persona-behaviour simulator, in: International Conference on Robot and Human Interactive Communication, IEEE. pp. 1–8. doi:10.1109/RO-MAN46459.2019.8956357. Angelopoulos,G.,Mangiacapra,L.,Rossi,A.,DiNapoli,C.,Rossi,S.,2025. Whatisbehindthecurtain?Increasingtransparencyinreinforcement learning with hum...

    work page doi:10.1109/ro-man46459.2019.8956357 2019

  3. [3]

    What will you do next?

    “What will you do next?" Designing and evaluating explanation generation using behavior trees for projection-level XAI, in: Companion of the International Conference on Human-Robot Interaction, ACM/IEEE. pp. 223–227. Love et al.:Preprint submitted to Elsevier Page 21 of 23 Temporal Counterfactual Explanations of Behaviour Tree Decisions doi:10.1145/361097...

    work page doi:10.1145/3610978.3640547

  4. [4]

    Robotics and Automation Letters 6, 5929–5936

    On the implementation of behavior trees in robotics. Robotics and Automation Letters 6, 5929–5936. doi:10.1109/LRA.2021.3087442. Colledanchise, M., Ögren, P.,

    work page doi:10.1109/lra.2021.3087442 2021

  5. [5]

    How behavior trees modularize robustness and safety in hybrid systems, in: International Conference on Intelligent Robots and Systems, IEEE/RSJ. pp. 1482–1488. doi:10.1109/IROS.2014.6942752. Colledanchise, M., Ögren, P.,

    work page doi:10.1109/iros.2014.6942752 2014

  6. [6]

    CRC Press

    Behavior trees in robotics and AI: An introduction. CRC Press. doi:10.1201/9780429489105. Cruz, F., Dazeley, R., Vamplew, P., Moreira, I.,

    work page doi:10.1201/9780429489105

  7. [7]

    Neural Computing and Applications 35, 18113–18130

    Explainable robotic systems: Understanding goal-driven actions in a reinforcement learning scenario. Neural Computing and Applications 35, 18113–18130. doi:10.1007/s00521-021-06425-5. DeChant, C.,

    work page doi:10.1007/s00521-021-06425-5

  8. [8]

    Robotics and Automation Letters 7, 8925–8932

    Why did I fail? A causal-based method to find explanations for robot failures. Robotics and Automation Letters 7, 8925–8932. doi:10.1109/LRA.2022.3188889. Doshi-Velez, F., Kim, B.,

    work page doi:10.1109/lra.2022.3188889 2022

  9. [9]

    Towards A Rigorous Science of Interpretable Machine Learning

    Towards a rigorous science of interpretable machine learning. arXiv preprint arXiv:1702.08608 doi:10.48550/ arXiv.1702.08608. Frering, L., Steinbauer-Wagner, G., Holzinger, A.,

    work page internal anchor Pith review Pith/arXiv arXiv

  10. [10]

    A surrogate model framework for explainable autonomous behaviour

    A surrogate model framework for explainable autonomous behaviour. arXiv preprint arXiv:2305.19724 doi:10.48550/arXiv.2305.19724. Gebellí, F., Hriscu, L., Ros, R., Lemaignan, S., Sanfeliu, A., Garrell, A.,

    work page doi:10.48550/arxiv.2305.19724

  11. [11]

    Personalised explainable robots using LLMs, in: International Conference on Human-Robot Interaction, ACM/IEEE. pp. 1304–1308. doi:10.1109/HRI61500.2025.10974125. Gjærum, V.B., Strümke, I., Lekkas, A.M., Miller, T.,

    work page doi:10.1109/hri61500.2025.10974125 2025

  12. [12]

    IFAC-PapersOnLine 56, 7–12

    Real-time counterfactual explanations for robotic systems with multiple continuous outputs. IFAC-PapersOnLine 56, 7–12. doi:10.1016/j.ifacol.2023.10.1328. Gómez-Talal, I., Azizsoltani, M., Bote-Curiel, L., Rojo-Álvarez, J.L., Singh, A.,

    work page doi:10.1016/j.ifacol.2023.10.1328 2023

  13. [13]

    Engineering Applications of Artificial Intelligence 155, 110664

    Towards explainable artificial intelligence in machine learning: A study on efficient perturbation-based explanations. Engineering Applications of Artificial Intelligence 155, 110664. Gugliermo,S.,Domínguez,D.C.,Iannotta,M.,Stoyanov,T.,Schaffernicht,E.,2024. Evaluatingbehaviortrees. RoboticsandAutonomousSystems 178, 104714. doi:10.1016/j.robot.2024.104714...

    work page doi:10.1016/j.robot.2024.104714 2024

  14. [14]

    Transactions on Human-Robot Interaction 10, 1–31

    Building thefoundation of robotexplanation generation using behavior trees. Transactions on Human-Robot Interaction 10, 1–31. doi:10.1145/3457185. Hayes,B.,Scassellati,B.,2016. Autonomouslyconstructinghierarchicaltasknetworksforplanningandhuman-robotcollaboration,in:2016IEEE International Conference on Robotics and Automation (ICRA), IEEE. pp. 5469–5476. ...

    work page doi:10.1145/3457185 2016

  15. [15]

    Oxford University Press

    Serial recall, in: The Oxford Handbook of Human Memory, Two Volume Pack: Foundations and Applications. Oxford University Press. chapter 28, pp. 799–830. doi:10.1093/oxfordhb/9780190917982.001.0001. Iovino, M., Förster, J., Falco, P., Chung, J.J., Siegwart, R., Smith, C.,

    work page doi:10.1093/oxfordhb/9780190917982.001.0001

  16. [16]

    arXiv preprint arXiv:2405.16137 doi:10.48550/arXiv.2405.16137

    Comparison between behavior trees and finite state machines. arXiv preprint arXiv:2405.16137 doi:10.48550/arXiv.2405.16137. Iovino, M., Scukins, E., Styrud, J., Ögren, P., Smith, C.,

    work page doi:10.48550/arxiv.2405.16137

  17. [17]

    Robotics and Autonomous Systems 154, 104096

    A survey of behavior trees in robotics and AI. Robotics and Autonomous Systems 154, 104096. doi:10.1016/j.robot.2022.104096. Kanazawa,K.,Koller,D.,Russell,S.,1995. Stochasticsimulationalgorithmsfordynamicprobabilisticnetworks,in:ConferenceonUncertaintyin Artificial Intelligence, pp. 346–351. doi:10.5555/2074158.2074197. Langley, P., Meadows, B., Sridharan...

    work page doi:10.1016/j.robot.2022.104096 2022

  18. [18]

    Explainable agency for intelligent autonomous systems, in: Conference on Artificial Intelligence, AAAI. pp. 4762–4763. doi:10.1609/aaai.v31i2.19108. LeMasurier,G.,Gautam,A.,Han,Z.,Crandall,J.W.,Yanco,H.A.,2024a.Reactiveorproactive?Howrobotsshouldexplainfailures,in:International Conference on Human-Robot Interaction, ACM/IEEE. pp. 413–422. doi:10.1145/3610...

    work page doi:10.1609/aaai.v31i2.19108 2024

  19. [19]

    Miller,T.,2019

    What would I do if...? Promoting understanding in HRI through real-time explanations in the wild, in: InternationalConferenceonRobotandHumanInteractiveCommunication,IEEE.pp.504–509.doi: 10.1109/RO-MAN60168.2024.10731403. Miller,T.,2019. Explanationinartificialintelligence:Insightsfromthesocialsciences. Artificialintelligence267,1–38. doi: 10.1016/j.artint...

    work page doi:10.1109/ro-man60168.2024.10731403 2024

  20. [20]

    T., Singh, S., & Guestrin, C

    From anecdotal evidence to quantitative evaluation methods: A systematic review on evaluating explainable AI. Computing Surveys 55, 1–42. doi:10.1145/3583558. Pearl, J., Mackenzie, D.,

    work page doi:10.1145/3583558

  21. [21]

    Basic books

    The book of why: the new science of cause and effect. Basic books. Potteiger,N.,Koutsoukos,X.,2023. Safeexplainableagentsforautonomousnavigationusingevolvingbehaviortrees,in:InternationalConference on Assured Autonomy, IEEE. pp. 44–52. doi:10.1109/ICAA58325.2023.00014. Russell, S., Norvig, P.,

    work page doi:10.1109/icaa58325.2023.00014 2023

  22. [22]

    Computing Surveys 55, 1–41

    Explainable goal-driven agents and robots-a comprehensive review. Computing Surveys 55, 1–41. doi:10.1145/3564240. Stonier,D.,Demir,E.M.,Walker,N.,Castro,S.,etal.,2025. Pytrees2.3.0. URL: https://py-trees.readthedocs.io/en/release-2.3.x/. Tagliamonte, C., Maccaline, D., LeMasurier, G., Yanco, H.A.,

    work page doi:10.1145/3564240 2025

  23. [23]

    A generalizable architecture for explaining robot failures using behavior trees and large language models, in: Companion of the International Conference on Human-Robot Interaction, ACM/IEEE. pp. 1038–1042. Love et al.:Preprint submitted to Elsevier Page 22 of 23 Temporal Counterfactual Explanations of Behaviour Tree Decisions doi:10.1145/3610978.3640551. ...

    work page doi:10.1145/3610978.3640551 2024

  24. [24]

    Unraveling explainable reinforcement learning using behavior tree structures, in: International Conference on Acoustics, Speech and Signal Processing, IEEE. pp. 6465–6469. doi:10.1109/ICASSP48485.2024.10446357. Love et al.:Preprint submitted to Elsevier Page 23 of 23

    work page doi:10.1109/icassp48485.2024.10446357 2024